SPEC OSG SPECmail2009 Benchmark
Workload Characterization for SPECmail_Ent2009 Metric

Mike Abbott, Yun-seng Chao

December 2008

 

Summary

This document summarizes the studies on mail server workload collected from multiple university and corporate sources, using a variety of IMAP4 clients. The analyzed workloads consist of both SMTP and IMAP4 requests. Each request is described by parameters which fully characterize its behavior. The proposed models, which are obtained by analyzing these parameters, are able to reproduce the behavior of the mail server workloads.

Document Organization

The report is organized as follows. We start with a description of the measurements and of the parameters considered in our studies. We then present the models characterizing the mail server workloads and we briefly describe how to use these models.

 

SPECmail2009 Additions/Changes

Much of the document discusses the workload changes between new SPECmail2009 and the original SPECmail2008 benchmark workload. Many of the internal distributions were updated with complete message and folder profiles provided by Apple, Inc in 2008. Most of this data replaces the original message and mailbox composition distributions. The SMTP traffic levels have been incorporated into the recipient and message size distributions.

One workload addition not discussed in this document is the ability to test using encrypted TCP connections. The reason lies in where this encryption incurs its cost. The e-mail clients issue commands according to user or programatic directives, regardless of the network connection's encryption mode. Empirical data shows both SUT and e-mail clients require extra computing and/or memory resources if encryption exists. Therefore, the benchmark's Secure metric influences the number of concurrent network sessions and interarrival times but not the actual command sequences. The two SPECmail2009 metrics show the effects of encrypted network connections on the SUT.

Measurements and Parameters

The measurements analyzed in our studies come from different sources. The measurements related to SMTP and IMAP4 have been provided by four companies and by two universities.? The collected sessions were divided into five IMAP4 and two SMTP groups.? The sessions within each group form the basis for all of the parameters that define the Enterprise User Profile, emulated by the SPECmail2009 benchmark.

 

Mailbox and Message Structures

The IMAP4 protocol allows email clients to create and maintain any number of folders and subfolders, in addition to the standard Inbox folder used in the SPECmail2001 POP3 user profile.? The IMAP4 command set also allows email clients to ask the server to describe these structures.? This information is independent of the delivery or retrieval protocols and so is treated outside of specific protocol and/or server context.

Multipurpose Internet Mail Extension (MIME) Profile

MIME is an internet attachment scheme, defined as a formal standard by RFCs 1521, 1522, and 1523.? The Sun and Apple data sets provided detailed information about mailbox and message structure.? Thus they form the basis for the following probability distribution tables used in the benchmark.?

The initial processing of all message sizes distinguished between single part sizes and multipart sizes.? The IMAP4 benchmark prioritizes individual MIME part size over the global message size distribution.

Single Part messages (Sun: 76% of total, Apple: 47% of total)

1. Use “Content-type: text/plain” or no content-type at all in message headers
2. Use subpart content size distribution

 

Multipart Message (Sun: 24% of total, Apple: 53% of total)

1. Use “Content Type: multipart/mixed; boundary=”xxxxxxxxx-counter” or “Content Type: multipart/alternative; boundary=”xxxxxxxxx-counter” in message headers
2. Use distributions for message part width and depth to help establish the set of multipart message bodies.
3. Categorize MIME messages to fall into one of these pre-defined multipart buckets.
4. Use subpart content size distribution to define the sub-part sizes in the fixed pool of pre-defined multipart messages.

Below are the distributions used in constructing messages in compliant with the MIME standard.

 

 

The following tables show the distribution of the number of MIME parts at the top level (without regard to nesting). It reflects the count of multipart/mixed parts immediately “attached” to the main message. It does not reflect any counting of multipart/alternative parts (i.e. text/plain and text/html, alternative formats of the same text). Nor does it reflect the MIME attachment depths (“attachments” to “attachments” or forwarded messages).

 

 

 

 

The next tables show the distribution of the nested MIME Part Levels that occur within a given message from the sample of MIME parts. It generally reflects messages or attachments which are forwarded multiple times, each time adding another depth level to the resulting message.

 

 

 

The following tables show the distribution of primary MIME Content Type (not including subtype) of all the parts in the entire sample.

 

 

After Sun's values were reviewed, a former employee noted that the Unix company that provided MIME distributions tended to use more text messages. Other companies have more and larger MIME parts that have richer, non-textual, content such as word processor documents, presentations, spreadsheets, web pages, calendar events, images, audio, and both rich and simple alternate MIME structures. The major effect of this shift is a tendency to increase the overall message sizes, and decreasing the Text content type in favor of the other categories.

However, increased Alternate structures does not eliminate the Text portion's counts. It just increases the other content types counters. Also, the IMAP server is not required to interpret the actual MIME parts content. It must extract the MIME part(s) and send the content, as is, to the IMAP4 client, which performs the interpretation. Therefore, the shift in Content Type distribution affects the benchmark's MIME structure of the message delivered to the SUT. The SUT still must deconstruct these MIME structures, but not the actual content.

 

Messages Per Folder

The following tables show the distribution of messages in folders at the first five levels.

 

Here is the same data from Apple bucketed such that each contains roughly five percentage points. These are the actual values used in the benchmark.

Mailbox Distribution Profile

 

A mail server that supports IMAP is likely to support a hierarchy of several mailboxes (also known folders) in addition to the default INBOX mailbox for each user.? Below are several distributions to construct the structure of mailboxes contained within a mailstore supported by IMAP.? The data used is extracted from the four enterprise data samples (Mirapoint, Openwave, Sun, Apple).

The following tables show the probably of an individual user having a certain number of mailboxes (aka folders) at each level (depth).? The data reflects the probability distributions for the first five (5) levels, even though the actual samples went many levels deeper than that.?

 

 

 

 

The following tables show the percent of folders at each level containing any subfolders.

 

 

 

Mailbox Structure Example

Below is a walk through of the construction of a folder tree with a diagram to illustrate the use of the above distribution tables in creating a folder tree for user “U1”. The probability values used are only examples, not actual distribution table entries.

 

The diagram below shows the mailbox structure for user U1.

Figure 1: Mailbox Structure Diagram

Peak Hour Determination

The overall peak traffic hour must be based on both SMTP and the corresponding IMAP activity over the same period of time.? Therefore, only two data samples were used to determine the relative workloads – Mirapoint and Openwave.? The other data samples did not provide corresponding SMTP logs for this purpose.

Peak Hour Traffic Volumes and Active Users

The following table shows the overall traffic volumes and users over the course of the peak day (determined by total number of message activity from the data samples).

 

 

The Peak Hour Percent of Active Users is computed by first using the larger of the two unique Sending or Recipient users and dividing that value by the total number of provisioned users.? The percentages are then pro-rated based on the relative number of actual users to compute the actual Percent of Active Users used in the benchmark.

 

 

PEAK_PCT_USERS = "78"

The Peak Hour Activity Percentage can be derived by using the traffic volume from the peak hour and the daily total for each protocol.? Again the benchmark value is computed by pro-rating each data sample within the overall user counts.

 

 

 

 

SMTP log files

SMTP Traffic Analysis

The SMTP log files reflect mail transfer agent workloads from the four enterprises, three collected over the course of fourteen (14) to thirty (30) days of operation and one from one day. The workload refers to all the requests processed by the mail server for delivering incoming and outgoing messages. These enterprises ranged from approximately 120 to 40,000 users.? The data logs cover the full 24-hour day, over the course of the data collection period.

The parameters used to describe the requests processed by the mail server are:

• time stamp of the request
• size [byte]
• number of recipients

The table below shows the statistics for SMTP traffic flows and message sizes for the four enterprises.?? The ISP user profile statistics are included to illustrate the difference with the original user model.

 

The following two tables contain the profile of the number of recipients per message, based on the Mirapoint and Openwave SMTP data from the busiest day of the week. The Apple SMTP sample did not distinguish between remote and local recipients, nor the actual recipient count from the RCPT TO step. So this version reuses the original recipient information. This data was extracted from both recipients named in the RCPT TO lines, as well as, recipient counts based on the mailing list expansions. The benchmark uses the probably distributions in the second table to generate the actual SMTP traffic.

 

 

 

 

 

 

 

As stated above, the recipient distribution includes traffic routed through mailing lists.? The data showed that 7-15% of overall SMTP traffic was sent to a mail distribution list.? This data is included in the recipient distributions above, but described below.? The benchmark does not require creation of any distribution lists.

 

 

 

 

SMTP Message Analysis

The SPECmail2001 method created a single level message that met a fixed message size distribution. Originally, SPECmail2009 attempted to follow the same criteria: generate messages according to the MIME distributions and then map these messages to the final SMTP derived message size distribution. However, these MIME distribution compliant messages did not comply with the SMTP log derived message size distribution. Analysis of the cause found the main reason for this size discrepency. The IMAP message samples form only a subset of the messages flowing through the e-mail system. The POP3 users on these same e-mail servers delete a substantial proportion of their messages, which does not keep messages on the e-mail server. Therefore, these missing messages did not contribute to the MIME definitions of the whole mail store.

Subsequent benchmark design choices prioritized message MIME parts structure and enclosure sizes over the derived SMTP message sizes. The SMTP log derived findings are presented here, but not used by the benchmark.

The SMTP log derived message size data differs between Consumer and Enterprise users. As with the earlier SPECmail2001, the size of each message is counted message size buckets. Overall, the average message size processed through the MTA increased from 24.5 KB to 38.57 KB, then to 101 KB.

The following tables describe the size distribution of all messages that flowed through the SMTP servers. This information is included here for completeness.

 

The SPECmail2009 Enterprise message size distribution has shifted towards larger values. The original enterprise sample's median was in the 2 KB size. The newer data sample exists between 4 and 8 KB. A majority continues to be messages slightly less than 8 KB in size. However, SPECmail2009 creates significantly more large messages. For all messages greater than 16 KB in size, SPECmail2008 created about 11.9% compared to the 37.8% created by SPECmail2009.

SMTP Message Rates

The corporate SMTP samples showed the following characteristics.

 

SMTP Normalized Profile (Peak Hour)

 

Workload models

We have built a model for each of the parameters characterizing the SMTP requests.

Inter-arrival time distribution

The message inter-arrival time computation uses a simplified model because the total number of messages tends not to be enough to fulfill a complex distribution.? Therefore, the time between message delivery is computed as the total number of messages to be delivered over the duration of the load test run time, divided by that run time.

Inter-arrival Time = (Number of Active Users) X (Messages per User) X (Recipients per Message) / Load Test Time (s)

 

Message Construction

As described in the message size and MIME parts analysis, the benchmark chose to follow message structural and attachment size distributions rather than the total message size distribution used by the earlier SPECmail2001 benchmark.? In that benchmark, the email server tends not to care about the actual message MIME structure and recognizes just headers versus body parts.? IMAP4 email clients understand the concepts of attachments and expect the e-mail server to understand the various message parts.? This meant that the e-mail server must evaluate the actual structure of each message.? Therefore, message structure and individual attachment sizes affect the actual message size, since the MIME structural description is embedded in the message but not visible to most users.

The benchmark uses the above MIME Parts, MIME Part Sizes and MIME Depth distribution tables to construct each message stored in the mail store.

MIME_PART_SIZE = "64,0.40%; 128,5.18%; 256,2.28%; 512,6.37%; 1024,9.22%; 2048,18.00%; 4096,28.97%; 8192,11.37%; 16384,6.46%; 32768,3.91%; 65536,3.02%; 131072,1.88%; 262144,1.21%; 524288,0.68%; 1048576,0.45%; 2097152,0.60%"

Number of recipients

Unlike the Consumer ISP user model, the Enterprise user model Number of Recipients Per Message for is not overwhelmingly dominated by a single value.? The effects of the internal distribution lists shifted the mean (5) away from the median (1) value.? Also, mail distribution lists tend to be used inside enterprises, the maximum recipient count was allowed to be at one hundred (100).

MSG_RECP_DISTRIBUTION = "1,75.11%; 2,8.03%; 3,6.08%; 4,1.59%; 5,1.10%; 6,1.48%; 7,0.61%; 8,0.40%; 9,0.34%; 10,0.30%; 15,2.44%; 20,0.69%; 25,0.57%; 50,0.69%; 100,0.39%; 500,0.18%"

How to use the models

The models described in the previous sections can be used to reproduce the behavior of the real workload of a mail server. In particular, the use of these models is based on sampling the various distributions identified for each of the three characterizing parameters.

To sample the Weibull distribution obtained for the inter-arrival times, it is necessary to invert the function and to derive the inter-arrival time from the probability distribution. To make it short, let u denote a random number uniformly distributed between 0 and 1, the inter-arrival time t _i between the i-th and (i-1)-th request is given by:

                       t _i = a * (- log (u) ) ^1/b            (***)

where log denotes the natural logarithm,  a and b are the parameters of the Weibull function. The procedure should then start by drawing a random number u and by computing the corresponding value of t using the previous formula. Note that u should be strictly greater than 0. For u=1, the value of the inter-arrival time is equal to zero.

In the case of message size and number of recipients, it is necessary to sample the distribution obtained from the buckets. Again it is necessary to draw a random number uniformly distributed between 0 and 1.
 

Scaling issues

Scaling issues arise when the workload model has to be used to represent the load of mail servers characterized by a smaller or larger number of users. This is particularly the case of the arrivals of the requests whose rate depends on the number of users of the mail server. The SPECMail2001 benchmark assumed a linear behavior of the arrival rates, that is, the arrival rate of the requests of a mail server with 100,000 users is 10 times the arrival rate of a mail server with 10,000 users. However, the SPECMail2009 benchmark will never approach the lower limits mandated in the previous model.? There are just not that many enterprises with 10,000 employees, much less 100,000.? A second consideration is that the number of new messages arriving during the peak hour is only a very small portion of the each user’s mail store.?

Therefore, the number of messages arriving for each user is spread at regular intervals.? The overall SMTP work load is determined by the message count multiplied by the number of users, and divided by sixty (60) seconds.?

IMAP4 log files

The IMAP data have been collected from various mail servers at the University of Wollongong, Purdue University, Mirapoint, Openwave and Sun.? The measurements were collected for fourteen (14) to thirty (30) days.? From these log files, IMAP commands from individual IMAP sessions were grouped together and analyzed.

The parameters used to describe each IMAP session are:

• time stamp
• number of messages and mailbox size
• number and size of messages retrieved/deleted within a session

The table below shows the IMAP command percentages generated during each data sample’s peak hour.

The statistics refer to one working day (24 hours); we have observed that the behavior of the users is very different over week-end days.

IMAP Session Model

Analysis of the 10’s of thousands of individual IMAP sessions led to the conclusion that different IMAP e-mail clients use the different interaction models.? Furthermore, these models were more complex than the POP3 sessions.? Each IMAP e-mail client could initiate from one (1) to five (5) concurrent session(s), each with its own distinct session initiation patterns.?

 

This leads to a complex IMAP Session model, defined by the combination of two categories: client-type and command sequences.? A command-sequence is a series of IMAP commands performing one or more mail operations within a specific session.? A client-type is a collection of one or more command-sequences.???

 

The following table describe the criteria for each command-sequence.

 

 

IMAP4 clients will use one or more of the five (5) command sequences.? The IMAP4 benchmark emulates four (4) client types.? During the benchmark run, each of these client type threads represents a single user.? A client may connect one or more times to the IMAP servers.

 

The compliant run uses the following combination to determine sequencing and dependencies.

CLIENT_TYPE_DISTRIBUTION = "1,31.373%; 3,32.353%; 4,3.922%; 5,2.941%; 13,3.922%; 14,10.784%; 15,1.961%; 24,0.980%; 34,2.941%; 45,2.941%; 134,0.980%; 145,3.922%; 1245,0.980%"

Each tuple defines the command sequence grouping (1 == CS1, 34 == CS3+CS4), and the percentage of overall load generator client threads that will implement each combination. The number of IMAP sessions varies as this matrix changes. Each load generator thread is assigned one specific combination.

IMAP Sample Selection

The extracted IMAP sessions were categorized according to command sequence types. However, some command sequences had an enormous number of sessions while others command sequence sample counts corresponded to the number of users. The fact that each client type uses more than one command sequence also forces the IMAP session selection criteria to gather all related sessions.

The final selection criteria used all IMAP sessions grouped by IMAP user name (found in each sessions’ login state). The resulting data set provides a more coherent model of not only individual primary command sequences (the premise of the SPECMail2001 benchmark) but also the number of related IMAP sessions and actions.

IMAP Command States

The IMAP command set allows many combinations of parameters and options. This means that a single IMAP command can perform more than one logical task, and on one or more messages at the same time. The best example is the FETCH command and its variant, UID FETCH. This single command has been used to retrieve not only the message body, but also message meta-data, headers and as a means to probe a folder for new messages. The latter (folder probe) is also complemented by the IMAP STATUS command, which provides a summary of old/new/deleted message.

The versatility of the IMAP command set leads to a need to expand the concept of a state from a simple command to the specific combination of a command and its parameters.? Included in this combination is the understanding of the number of messages encompassed by that command state, as well as whether it is against an individual, contiguous series or a disjoint set of messages.

IMAP State Codes tracked by Mail2009 Benchmark

State Identfier	State Name
1.                   ?	APPEND
2.	CHECK
3.	CLOSE
4.	COPY_NUM_FOLDER
5.	COPY_RANGE_FOLDER
6.	CREATE
7.	DELETE
8.	EXAMINE_FOLDER
9.	EXAMINE_INBOX
10.	EXAMINE_INBOXSENT
11.	EXAMINE_SENT
12.	EXAMINE_SENT_ITEMS
13.	EXPUNGE
14.	FETCH_NUM
15.	FETCH_NUM_BODYALL
16.	FETCH_NUM_BODYPARTS
17.	FETCH_NUM_BODYPEEK
18.	FETCH_NUM_BODYPEEK_HEADER
19.	FETCH_NUM_BODYPEEK_HEADERFIELDS
20.	FETCH_NUM_BODYSTRUCTURE_FLAGS
21.	FETCH_NUM_BODY_BODYALL_HEADERFIELDS
22.	FETCH_NUM_BODY_HEADER
23.	FETCH_NUM_ENVELOPE_BODYPEEK_HEADERFIELDS_BODYSTRUCTURE_FLAGS_INTERNALDATE_RFC822SIZE
24.	FETCH_NUM_ENVELOPE_BODYPEEK_HEADERFIELDS_FLAGS_INTERNALDATE_RFC822SIZE_UID
25.	FETCH_NUM_FLAGS
26.	FETCH_NUM_FLAGS_BODYPEEK_HEADERFIELDS_INTERNALDATE_RFC822SIZE
27.	FETCH_NUM_FLAGS_BODYSTRUCTURE_ENVELOPE_INTERNALDATE_RFC822SIZE_UID
28.	FETCH_NUM_RFC822HEADER
29.	FETCH_NUM_RFC822TEXT
30.	FETCH_NUM_UID
31.	FETCH_NUM_UID_BODYPEEK_HEADERFIELDS_ENVELOPE_FLAGS_INTERNALDATE_RFC822SIZE
32.	FETCH_RANGE_UID
33.	FETCH_RANGE_BODYPEEK_HEADERFIELDS
34.	FETCH_RANGE_ENVELOPE_BODYPEEK_HEADERFIELDS_FLAGS_INTERNALDATE_RFC822SIZE_UID
35.	FETCH_RANGE_FLAGS_BODYPEEK_HEADERFIELDS_INTERNALDATE_RFC822SIZE
36.	FETCH_RANGE_FLAGS_BODYSTRUCTURE_ENVELOPE_INTERNALDATE_RFC822SIZE_UID
37.	FETCH_RANGE_UID_BODYPEEK_HEADERFIELDS_ENVELOPE_FLAGS_INTERNALDATE_RFC822SIZE
38.	FETCH_SERIES_ENVELOPE_BODYPEEK_HEADERFIELDS_FLAGS_INTERNALDATE_RFC822SIZE_UID
39.	FETCH_SERIES_ENVELOPE_BODYSTRUCTURE_INTERNALDATE_RFC822SIZE
40.	FETCH_SERIES_FLAGS_BODYPEEK_HEADERFIELDS_INTERNALDATE_RFC822SIZE
41.	FETCH_SERIES_UID
42.	FETCH_UID
43.	LIST
44.	LOGIN
45.	LOGOUT
46.	LSUB_NULL_FOLDER
47.	LSUB_NULL_PART
48.	LSUB_NULL_SENT
49.	LSUB_NULL_WILDCARD
50.	LSUB_WILDCARD_WILDCARD
51.	NOOP
52.	RENAME_FOLDER_FOLDER
53.	RENAME_INBOXINBOXSENT_INBOXTRASHINBOXSENT
54.	SEARCH_ALL_DELETED
55.	SEARCH_ALL_RANGE_CHARSET_RFCHEADER
56.	SEARCH_ALL_RFCHEADER
57.	SEARCH_ALL_UNDELETED_UNSEEN
58.	SEARCH_DELETED
59.	SEARCH_RFCHEADER
60.	SEARCH_UNDELETED
61.	SEARCH_UNSEEN
62.	SELECT_
63.	SELECT_FOLDER
64.	SELECT_FOLDER_ITEMS
65.	SELECT_INBOX
66.	SELECT_INBOXSENT
67.	SELECT_INBOXSENT_ITEMS
68.	SELECT_SENT
69.	SELECT_SENT_ITEMS
70.	STARTED
71.	STATUS_FOLDER_ITEMS_MESSAGES_UNSEEN
72.	STATUS_FOLDER_ITEMS_UNSEEN
73.	STATUS_FOLDER_MESSAGES
74.	STATUS_FOLDER_MESSAGES_RECENT_UNSEEN_UIDVALIDITY_UIDNEXT
75.	STATUS_FOLDER_MESSAGES_UNSEEN
76.	STATUS_FOLDER_UIDNEXT
77.	STATUS_FOLDER_UIDNEXT_UIDVALIDITY_MESSAGES
78.	STATUS_FOLDER_UNSEEN
79.	STATUS_INBOXSENT_ITEMS_MESSAGES_UNSEEN
80.	STATUS_INBOXSENT_ITEMS_UNSEEN
81.	STATUS_INBOXSENT_UNSEEN
82.	STATUS_INBOXSENT_MESSAGES_UNSEEN
83.	STATUS_INBOX_MESSAGES_RECENT_UNSEEN_UIDVALIDITY_UIDNEXT
84.	STATUS_INBOX_MESSAGES_UNSEEN
85.	STATUS_INBOX_UIDNEXT
86.	STATUS_INBOX_UIDNEXT_UIDVALIDITY_MESSAGES
87.	STATUS_INBOX_UNSEEN
88.	STATUS_SENT_ITEMS_MESSAGES_UNSEEN
89.	STATUS_SENT_ITEMS_UNSEEN
90.	STATUS_SENT_MESSAGES_UNSEEN
91.	STATUS_SENT_UNSEEN
92.	STORE_NUM_SET_FLAGS_ANSWERED
93.	STORE_NUM_SET_FLAGS_DELETED
94.	STORE_NUM_SET_FLAGS_SEEN
95.	STORE_NUM_UNSET_FLAGS_DELETED
96.	STORE_NUM_UNSET_FLAGS_SEEN
97.	STORE_RANGE_SET_FLAGS_DELETED
98.	STORE_RANGE_SET_FLAGS_SEEN
99.	STORE_SERIES_SET_FLAGS_DELETED
100.	STORE_UNTILEND_SET_FLAGS_DELETED
101.	STORE_UNTILEND_SET_FLAGS_SEEN
102.	SUBSCRIBE_FOLDER
103.	SUBSCRIBE_INBOXSENT
104.	UID_COPY_NUM_FOLDER
105.	UID_COPY_NUM_INBOX
106.	UID_COPY_NUM_INBOXSENT
107.	UID_COPY_RANGE_FOLDER
108.	UID_COPY_RANGE_INBOX
109.	UID_COPY_RANGE_INBOXSENT
110.	UID_COPY_SERIES_FOLDER
111.	UID_FETCH_NUM_BODY
112.	UID_FETCH_NUM_BODYALL
113.	UID_FETCH_NUM_BODYPARTS
114.	UID_FETCH_NUM_BODYPEEK
115.	UID_FETCH_NUM_BODYPEEKALL
116.	UID_FETCH_NUM_BODYPEEK_HEADER
117.	UID_FETCH_NUM_BODYPEEK_UID
118.	UID_FETCH_NUM_BODYSTRUCTURE
119.	UID_FETCH_NUM_BODY_BODYMIMEALL_BODYMIMEPARTS_HEADER
120.	UID_FETCH_NUM_BODY_BODYMIMEALL_HEADER
121.	UID_FETCH_NUM_BODY_HEADER
122.	UID_FETCH_NUM_ENVELOPE
123.	UID_FETCH_NUM_FLAGS
124.	UID_FETCH_NUM_RFC822SIZE
125.	UID_FETCH_NUM_UID
126.	UID_FETCH_NUM_UID_BODYPEEK_FLAGS_INTERNALDATE
127.	UID_FETCH_NUM_UID_BODYPEEK_FLAGS_INTERNALDATE_RFC822SIZE
128.	UID_FETCH_NUM_UID_BODYPEEK_HEADERFIELDS_FLAGS_RFC822SIZE
129.	UID_FETCH_NUM_UID_BODYPEEK_HEADER_FLAGS_INTERNALDATE_RFC822SIZE
130.	UID_FETCH_NUM_UID_BODYPEEK_RFC822SIZE
131.	UID_FETCH_NUM_UID_BODY_RFC822SIZE
132.	UID_FETCH_RANGE_UID_BODYPEEK_FLAGS_INTERNALDATE
133.	UID_FETCH_RANGE_UID_BODYPEEK_HEADERFIELDS_FLAGS_RFC822SIZE
134.	UID_FETCH_RANGE_UID_BODYPEEK_RFC822SIZE
135.	UID_FETCH_RANGE_UID_ENVELOPE_FLAGS_INTERNALDATE_RFC822SIZE
136.	UID_FETCH_RANGE_UID_FLAGS
137.	UID_FETCH_RANGE_UID_RFC822SIZE_BODYPEEK_HEADERFIELDS
138.	UID_FETCH_RANGE_UID_UID_BODYPEEK_HEADER_HEADERFIELDS_FLAGS_FLAGS_RFC822SIZE_RFC822SIZE_UID
139.	UID_FETCH_SERIES_UID_BODYPEEK_FLAGS_INTERNALDATE
140.	UID_FETCH_SERIES_UID_BODYPEEK_HEADERFIELDS_FLAGS_RFC822SIZE
141.	UID_FETCH_SERIES_UID_BODYPEEK_RFC822SIZE
142.	UID_FETCH_UID_BODYPEEK_HEADERFIELDS_FLAGS_RFC822SIZE
143.	UID_FETCH_UID_BODYPEEK_HEADER_FLAGS_INTERNALDATE_RFC822SIZE
144.	UID_FETCH_UNTILEND_BODYPEEK_HEADERFIELDS_ENVELOPE_FLAGS_INTERNALDATE_RFC822SIZE_UID
145.	UID_FETCH_UNTILEND_ENVELOPE_FLAGS_INTERNALDATE_RFC822SIZE_UID
146.	UID_FETCH_UNTILEND_FLAGS
147.	UID_FETCH_UNTILEND_UID_BODYPEEK_HEADERFIELDS_FLAGS_RFC822SIZE
148.	UID_FETCH_UNTILEND_UID_BODYPEEK_HEADER_FLAGS_INTERNALDATE_RFC822SIZE
149.	UID_FETCH_UNTILEND_UID_FLAGS
150.	UID_FETCH_UNTILEND_UID_FLAGS_INTERNALDATE_RFC822HEADER_RFC822SIZE
151.	UID_SEARCH_ANSWERED
152.	UID_SEARCH_DELETED
153.	UID_SEARCH_FLAGGED
154.	UID_SEARCH_HEADER_QUESTION_RFCHEADER_UNDELETED
155.	UID_SEARCH_HEADER_RFCHEADER_UNDELETED
156.	UID_SEARCH_HEADER_UNDELETED
157.	UID_SEARCH_KEYWORD
158.	UID_SEARCH_NOTDELETED_UID_UNTILEND
159.	UID_SEARCH_RFCHEADER_UNDELETED
160.	UID_SEARCH_SEEN
161.	UID_SEARCH_SINCE
162.	UID_SEARCH_UID_NUM
163.	UID_SEARCH_UID_NUM_NOTDELETED
164.	UID_SEARCH_UID_RANGE
165.	UID_SEARCH_UID_RANGE_NOTDELETED
166.	UID_SEARCH_UID_UNTILEND_UNDELETED_UNDRAFT_UNSEEN
167.	UID_SEARCH_UID_UNTILEND_UNDELETED_UNSEEN
168.	UID_SEARCH_UNDELETED
169.	UID_SEARCH_UNDELETED_UNSEEN
170.	UID_SEARCH_UNSEEN
171.	UID_SEARCH_UNTILEND
172.	UID_STORE_NUM_SET_FLAGS_ANSWERED
173.	UID_STORE_NUM_SET_FLAGS_ANSWERED_DELETED_SEEN
174.	UID_STORE_NUM_SET_FLAGS_ANSWERED_SEEN
175.	UID_STORE_NUM_SET_FLAGS_DELETED
176.	UID_STORE_NUM_SET_FLAGS_DELETED_SEEN
177.	UID_STORE_NUM_SET_FLAGS_FLAGGED
178.	UID_STORE_NUM_SET_FLAGS_SEEN
179.	UID_STORE_NUM_SET_FLAGS_SEEN_ANSWERED
180.	UID_STORE_NUM_SET_FLAGS_SEEN_DELETED
181.	UID_STORE_NUM_UNSET_FLAGS
182.	UID_STORE_NUM_UNSET_FLAGS_ANSWERED
183.	UID_STORE_NUM_UNSET_FLAGS_DELETED
184.	UID_STORE_NUM_UNSET_FLAGS_FLAGGED
185.	UID_STORE_NUM_UNSET_FLAGS_FLAGGED_ANSWERED
186.	UID_STORE_NUM_UNSET_FLAGS_FLAGGED_FORWARDED_MDNSENT_DELETED_DRAFT
187.	UID_STORE_NUM_UNSET_FLAGS_SEEN
188.	UID_STORE_NUM_UNSET_FLAGS_SEEN_ANSWERED
189.	UID_STORE_NUM_UNSET_FLAGS_SEEN_ANSWERED_DELETED
190.	UID_STORE_NUM_UNSET_FLAGS_SEEN_ANSWERED_DELETED_DRAFT_FLAGGED
191.	UID_STORE_NUM_UNSET_FLAGS_SEEN_ANSWERED_DELETED_FLAGGED
192.	UID_STORE_NUM_UNSET_FLAGS_SEEN_ANSWERED_FLAGGED
193.	UID_STORE_NUM_UNSET_FLAGS_SEEN_DELETED
194.	UID_STORE_NUM_UNSET_FLAGS_SEEN_FLAGGED
195.	UID_STORE_NUM_UNSET_FLAGS_SEEN_FORWARDED_MDNSENT_ANSWERED_DELETED_DRAFT_FLAGGED
196.	UID_STORE_NUM_UNSET_FLAGS_SEEN_FORWARDED_MDNSENT_DELETED_DRAFT_FLAGGED
197.	UID_STORE_NUM_UNSET_FLAGS_SEEN_MDNSENT_ANSWERED_DELETED_DRAFT_FLAGGED
198.	UID_STORE_RANGE_SET_FLAGS_ANSWERED
199.	UID_STORE_RANGE_SET_FLAGS_DELETED
200.	UID_STORE_RANGE_SET_FLAGS_DELETED_SEEN
201.	UID_STORE_RANGE_SET_FLAGS_SEEN
202.	UID_STORE_RANGE_SET_FLAGS_SEEN_DELETED
203.	UID_STORE_RANGE_UNSET_FLAGS
204.	UID_STORE_RANGE_UNSET_FLAGS_ANSWERED_FORWARDED_MDNSENT_DELETED_DRAFT_FLAGGED
205.	UID_STORE_RANGE_UNSET_FLAGS_DELETED
206.	UID_STORE_RANGE_UNSET_FLAGS_SEEN
207.	UID_STORE_RANGE_UNSET_FLAGS_SEEN_FORWARDED_MDNSENT_ANSWERED_DELETED_DRAFT_FLAGGED
208.	UID_STORE_SERIES_SET_FLAGS_DELETED
209.	UID_STORE_SERIES_SET_FLAGS_DELETED_SEEN
210.	UID_STORE_SERIES_SET_FLAGS_SEEN
211.	UID_STORE_SERIES_UNSET_FLAGS_SEEN_FORWARDED_MDNSENT_ANSWERED_DELETED_DRAFT_FLAGGED
212.	UID_STORE_UNSET_FLAGS_SEEN
213.	UNSUBSCRIBE_FOLDER
214.	SEARCH_ALL_CALL_INFORMATION
215.	UID_COPY_NUM_
216.	UID_COPY_NUM_TRASH
217.	UID_COPY_RANGE_TRASH
218.	UID_COPY_SERIES_TRASH
219.	UID_FETCH_NUM_BODYPEEK_RFC822SIZE_UID
220.	UID_FETCH_NUM_BODY_RFC822SIZE_UID
221.	UID_FETCH_NUM_UID_BODYPEEK_HEADER_FLAGS_RFC822SIZE
222.	UID_FETCH_RANGE_UID_BODYPEEK_HEADER_FLAGS_RFC822SIZE
223.	LSUB
224.	SUBSCRIBE_TRASH
225.	UID_COPY_NUM
226.	UID_COPY_RANGE
227.	UID_COPY_SERIES
228.	UID_FETCH_NUM_BODYMIMEALL
229.	UID_FETCH_NUM_UID_BODYSTRUCTURE
230.	UID_FETCH_RANGE_BODYPEEK_HEADERFIELDS
231.	UID_FETCH_UNTILEND_FLAGS_RFC822SIZE
232.	SESSION_START

 

Functionally, there are many redundant states.? However, it was felt that the effort to use message unique identifier (UID) versus the variable message index number is significant. The UID remains fixed for the life of the mailstore. The message index number changes as the number of messages change. This means a message has many relative index numbers across multiple IMAP sessions.  For similar reasons, it was felt that operating on a contiguous range of messages generates a different workload than a random set of message numbers.

The various FETCH commands against headers and body sections make more sense when divided according to command sequences and client types. Many of these are the other half of a folder probe, depending on the client type. The present or absence of MIME parts also factor into the construction of the final command, as these lead to further probes of individual MIME parts.

The actual state transition charts used by the benchmark is too complex for this document. The Architecture White Paper provides the named objects used in the SPECmail2009 source code.

Inter-arrival Time Distributions

The uncertainty or randomness associated with the arrival of IMAP commands to the server is modeled using a Markov model to specify the statistical relationships between commands as transition-probability matrices. Each command sequence has its own probabilities and probability distributions of specific states. All entries in these transition-probability matrices were derived from a subset of the data samples. The base inter-arrival transition matrix uses a lognormal formula similar to the one used for SPECMail2001. Please consult the source code for the actual values.

IMAP Scaling Considerations

Since each IMAP user represents a variable number of IMAP sessions and incurs a considerable amount of storage space, scaling has problems at both ends of the mandated range. A further complication is the actual distribution of command sequences, since some occur much more frequently than others.

A benchmark run with too few users runs into the problem of missing command sequences as well as compliance with the mail store folder structures, message size and count distributions. Experiments have shown that a minimum of 250 users must be used for a compliant run that meets folder and message structure distributions.

The other end of the scaling problem lies in the number of IMAP users and the fact that each user represents one or more concurrent IMAP session. The IMAP benchmark apparently supports fewer users per load generator, compared to SPECmail2001. But this is very misleading since the correct consideration should be the concurrent number of client sessions and their activity levels in these sessions.

The POP3 benchmark defines very short session times - on the order of a few seconds for at least 75% of all POP3 sessions that find no messages to retrieve. So dispite the large number of defined POP3 users, the 25% active users only log into the system four times during the peak hour. Furthermore, the typical POP3 session lasts only 2-5 seconds, executing at most 10 commands (25%), but usually three commands (75%) within each session. This means that only a small subset of users is actively connected at any one time.

In constrast, IMAP users are always connected and active. The number of IMAP users determines the minimum concurrent IMAP sessions that stay logged into the IMAP server for the entire peak hour simulation. The client type distribution values then determines the number of ancillary IMAP sessions that will be generated. This means the IMAP server should allow at least

4.5 X UserCount

concurrent client socket connections.