SideStream#

SideStream collected statistics about the TCP connections used by the measurement tools running on the M-Lab platform when it used the web100 kernel extension. SideStream ran as a “side” addition to other tools, taking advantage of TCP connections generated by incoming tests to M-Lab.

Source code#

https://github.com/npad/sidestream

Citing the M-Lab SideStream Dataset#

Please cite this data set as follows: The M-Lab SideStream Data Set, <date range used>. https://measurementlab.net/tests/sidestream

SideStream Data in Raw Format#

Data collected by SideStream is available in raw format in Google Cloud Storage: https://console.developers.google.com/storage/browser/archive-measurement-lab/sidestream/.

SideStream Data in BigQuery#

M-Lab parsed all SideStream data into BigQuery tables and views, and makes query access available for free by subscription to a Google Group. Find out more about how to get access on our BigQuery QuickStart page.

BigQuery Tables/Views for SideStream Data:

measurement-lab.sidestream.web100

Sidestream Schema#

The table below describes the schema for the table: measurement-lab.sidestream.web100.

Field name	Type	Description
`partition_date`	`date`	The day (in UTC) in which the data was loaded.
`test_id`	`string`	ID of the test. It represents the filename of the log that contains the data generated during the test (e.g. `20090819T02:01:04.507508000Z_189.6.232.77:3859.c2s_snaplog.gz`).
`task_filename`	`string`	The raw data file in Google Cloud Storage from which the test row was parsed.
`parse_time`	`timestamp`	The timestamp of when test data was parsed into BigQuery from Google Cloud Storage.
`parser_version`	`string`	A link to the tagged version of the M-Lab ETL parser which processed the test row.
`project`	`integer`
`log_time`	`timestamp`
`type`	`integer`
`anomalies`	`record`
`anomalies.exclusion_level`	`integer`
`web100_log_entry`	`record`
`web100_log_entry.log_time`	`integer`
`web100_log_entry.version`	`string`
`web100_log_entry.group_name`	`string`
`web100_log_entry.connection_spec.local_af`	`integer`	IPv4 = `0` IPv6 = `1`
`web100_log_entry.connection_spec.local_ip`	`string`
`web100_log_entry.connection_spec.local_port`	`integer`
`web100_log_entry.connection_spec.remote_ip`	`string`
`web100_log_entry.connection_spec.remote_port`	`integer`
`web100_log_entry.connection_spec.local_geolocation`	`record`
`web100_log_entry.connection_spec.local_geolocation.area_code`	`integer`
`web100_log_entry.connection_spec.local_geolocation.city`	`string`
`web100_log_entry.connection_spec.local_geolocation.continent_code`	`string`
`web100_log_entry.connection_spec.local_geolocation.country_code`	`string`
`web100_log_entry.connection_spec.local_geolocation.country_code3`	`string`
`web100_log_entry.connection_spec.local_geolocation.country_name`	`string`
`web100_log_entry.connection_spec.local_geolocation.latitude`	`float`
`web100_log_entry.connection_spec.local_geolocation.longitude`	`float`
`web100_log_entry.connection_spec.local_geolocation.metro_code`	`integer`
`web100_log_entry.connection_spec.local_geolocation.postal_code`	`string`
`web100_log_entry.connection_spec.local_geolocation.region`	`string`
`web100_log_entry.connection_spec.local_geolocation.radius`	`integer`
`web100_log_entry.connection_spec.remote_geolocation.area_code`	`integer`
`web100_log_entry.connection_spec.remote_geolocation`	`record`
`web100_log_entry.connection_spec.remote_geolocation.area_code`	`integer`
`web100_log_entry.connection_spec.remote_geolocation.city`	`string`
`web100_log_entry.connection_spec.remote_geolocation.continent_code`	`string`
`web100_log_entry.connection_spec.remote_geolocation.country_code`	`string`
`web100_log_entry.connection_spec.remote_geolocation.country_code3`	`string`
`web100_log_entry.connection_spec.remote_geolocation.country_name`	`string`
`web100_log_entry.connection_spec.remote_geolocation.latitude`	`float`
`web100_log_entry.connection_spec.remote_geolocation.longitude`	`float`
`web100_log_entry.connection_spec.remote_geolocation.metro_code`	`integer`
`web100_log_entry.connection_spec.remote_geolocation.postal_code`	`string`
`web100_log_entry.connection_spec.remote_geolocation.region`	`string`
`web100_log_entry.connection_spec.remote_geolocation.radius`	`integer`
`web100_log_entry.snap`	`record`
`web100_log_entry.snap.AbruptTimeouts`	`integer`
`web100_log_entry.snap.ActiveOpen`	`integer`
`web100_log_entry.snap.CERcvd`	`integer`
`web100_log_entry.snap.CongAvoid`	`integer`
`web100_log_entry.snap.CongOverCount`	`integer`
`web100_log_entry.snap.CongSignals`	`integer`
`web100_log_entry.snap.CountRTT`	`integer`
`web100_log_entry.snap.CurAppRQueue`	`integer`
`web100_log_entry.snap.CurAppWQueue`	`integer`
`web100_log_entry.snap.CurCwnd`	`integer`
`web100_log_entry.snap.CurMSS`	`integer`
`web100_log_entry.snap.CurRTO`	`integer`
`web100_log_entry.snap.CurReasmQueue`	`integer`
`web100_log_entry.snap.CurRetxQueue`	`integer`
`web100_log_entry.snap.CurRwinRcvd`	`integer`
`web100_log_entry.snap.CurRwinSent`	`integer`
`web100_log_entry.snap.CurSsthresh`	`integer`
`web100_log_entry.snap.CurTimeoutCount`	`integer`
`web100_log_entry.snap.DSACKDups`	`integer`
`web100_log_entry.snap.DataOctetsIn`	`integer`
`web100_log_entry.snap.DataOctetsOut`	`integer`
`web100_log_entry.snap.DataSegsIn`	`integer`
`web100_log_entry.snap.DataSegsOut`	`integer`
`web100_log_entry.snap.DupAckEpisodes`	`integer`
`web100_log_entry.snap.DupAcksIn`	`integer`
`web100_log_entry.snap.DupAcksOut`	`integer`
`web100_log_entry.snap.Duration`	`integer`
`web100_log_entry.snap.ECESent`	`integer`
`web100_log_entry.snap.ECN`	`integer`
`web100_log_entry.snap.ECNNonceRcvd`	`integer`
`web100_log_entry.snap.ECNsignals`	`integer`
`web100_log_entry.snap.ElapsedMicroSecs`	`integer`
`web100_log_entry.snap.ElapsedSecs`	`integer`
`web100_log_entry.snap.FastRetran`	`integer`
`web100_log_entry.snap.HCDataOctetsIn`	`integer`
`web100_log_entry.snap.HCDataOctetsOut`	`integer`
`web100_log_entry.snap.HCSumRTT`	`integer`
`web100_log_entry.snap.HCThruOctetsAcked`	`integer`
`web100_log_entry.snap.HCThruOctetsReceived`	`integer`
`web100_log_entry.snap.InRecovery`	`integer`
`web100_log_entry.snap.IpTosIn`	`integer`
`web100_log_entry.snap.IpTosOut`	`integer`
`web100_log_entry.snap.IpTtl`	`integer`
`web100_log_entry.snap.LimCwnd`	`integer`
`web100_log_entry.snap.LimMSS`	`integer`
`web100_log_entry.snap.LimRwin`	`integer`
`web100_log_entry.snap.LimSsthresh`	`integer`
`web100_log_entry.snap.LocalAddress`	`string`
`web100_log_entry.snap.LocalAddressType`	`integer`
`web100_log_entry.snap.LocalPort`	`integer`
`web100_log_entry.snap.MSSRcvd`	`integer`
`web100_log_entry.snap.MSSSent`	`integer`
`web100_log_entry.snap.MaxAppRQueue`	`integer`
`web100_log_entry.snap.MaxAppWQueue`	`integer`
`web100_log_entry.snap.MaxCaCwnd`	`integer`
`web100_log_entry.snap.MaxMSS`	`integer`
`web100_log_entry.snap.MaxPipeSize`	`integer`
`web100_log_entry.snap.MaxRTO`	`integer`
`web100_log_entry.snap.MaxRTT`	`integer`
`web100_log_entry.snap.MaxReasmQueue`	`integer`
`web100_log_entry.snap.MaxRetxQueue`	`integer`
`web100_log_entry.snap.MaxRwinRcvd`	`integer`
`web100_log_entry.snap.MaxRwinSent`	`integer`
`web100_log_entry.snap.MaxSsCwnd`	`integer`
`web100_log_entry.snap.MaxSsthresh`	`integer`
`web100_log_entry.snap.MinMSS`	`integer`
`web100_log_entry.snap.MinRTO`	`integer`
`web100_log_entry.snap.MinRTT`	`integer`
`web100_log_entry.snap.MinRwinRcvd`	`integer`
`web100_log_entry.snap.MinRwinSent`	`integer`
`web100_log_entry.snap.MinSsthresh`	`integer`
`web100_log_entry.snap.Nagle`	`integer`
`web100_log_entry.snap.NonRecovDA`	`integer`
`web100_log_entry.snap.NonRecovDAEpisodes`	`integer`
`web100_log_entry.snap.OctetsRetrans`	`integer`
`web100_log_entry.snap.OtherReductions`	`integer`
`web100_log_entry.snap.PipeSize`	`integer`
`web100_log_entry.snap.PostCongCountRTT`	`integer`
`web100_log_entry.snap.PostCongSumRTT`	`integer`
`web100_log_entry.snap.PreCongSumCwnd`	`integer`
`web100_log_entry.snap.PreCongSumRTT`	`integer`
`web100_log_entry.snap.QuenchRcvd`	`integer`
`web100_log_entry.snap.RTTVar`	`integer`
`web100_log_entry.snap.RcvNxt`	`integer`
`web100_log_entry.snap.RcvRTT`	`integer`
`web100_log_entry.snap.RcvWindScale`	`integer`
`web100_log_entry.snap.RecInitial`	`integer`
`web100_log_entry.snap.RemAddress`	`integer`
`web100_log_entry.snap.RemPort`	`integer`
`web100_log_entry.snap.RetranThresh`	`integer`
`web100_log_entry.snap.SACK`	`integer`
`web100_log_entry.snap.SACKBlocksRcvd`	`integer`
`web100_log_entry.snap.SACKsRcvd`	`integer`
`web100_log_entry.snap.SampleRTT`	`integer`
`web100_log_entry.snap.SegsIn`	`integer`
`web100_log_entry.snap.SegsOut`	`integer`
`web100_log_entry.snap.SegsRetrans`	`integer`
`web100_log_entry.snap.SendStall`	`integer`
`web100_log_entry.snap.SlowStart`	`integer`
`web100_log_entry.snap.SmoothedRTT`	`integer`
`web100_log_entry.snap.SndInitial`	`integer`
`web100_log_entry.snap.SndLimBytesCwnd`	`integer`
`web100_log_entry.snap.SndLimBytesRwin`	`integer`
`web100_log_entry.snap.SndLimBytesSender`	`integer`
`web100_log_entry.snap.SndLimTimeCwnd`	`integer`
`web100_log_entry.snap.SndLimTimeRwin`	`integer`
`web100_log_entry.snap.SndLimTimeSnd`	`integer`
`web100_log_entry.snap.SndLimTransCwnd`	`integer`
`web100_log_entry.snap.SndLimTransRwin`	`integer`
`web100_log_entry.snap.SndLimTransSnd`	`integer`
`web100_log_entry.snap.SndMax`	`integer`
`web100_log_entry.snap.SndNxt`	`integer`
`web100_log_entry.snap.SndUna`	`integer`
`web100_log_entry.snap.SndWindScale`	`integer`
`web100_log_entry.snap.SoftErrorReason`	`integer`
`web100_log_entry.snap.SoftErrors`	`integer`
`web100_log_entry.snap.SpuriousFrDetected`	`integer`
`web100_log_entry.snap.SpuriousRtoDetected`	`integer`
`web100_log_entry.snap.StartTimeStamp`	`integer`
`web100_log_entry.snap.State`	`integer`
`web100_log_entry.snap.SubsequentTimeouts`	`integer`
`web100_log_entry.snap.SumOctetsReordered`	`integer`
`web100_log_entry.snap.SumRTT`	`integer`
`web100_log_entry.snap.ThruOctetsAcked`	`integer`
`web100_log_entry.snap.ThruOctetsReceived`	`integer`
`web100_log_entry.snap.TimeStampRcvd`	`integer`
`web100_log_entry.snap.TimeStampSent`	`integer`
`web100_log_entry.snap.TimeStamps`	`integer`
`web100_log_entry.snap.Timeouts`	`integer`
`web100_log_entry.snap.WAD_CwndAdjust`	`integer`
`web100_log_entry.snap.WAD_IFQ`	`integer`
`web100_log_entry.snap.WAD_MaxBurst`	`integer`
`web100_log_entry.snap.WAD_MaxSsthresh`	`integer`
`web100_log_entry.snap.WAD_NoAI`	`integer`
`web100_log_entry.snap.WillSendSACK`	`integer`
`web100_log_entry.snap.WillUseSACK`	`integer`
`web100_log_entry.snap.WinScaleRcvd`	`integer`
`web100_log_entry.snap.WinScaleSent`	`integer`
`web100_log_entry.snap.X_OtherReductionsCM`	`integer`
`web100_log_entry.snap.X_OtherReductionsCV`	`integer`
`web100_log_entry.snap.X_Rcvbuf`	`integer`
`web100_log_entry.snap.X_Sndbuf`	`integer`
`web100_log_entry.snap.X_dbg1`	`integer`
`web100_log_entry.snap.X_dbg2`	`integer`
`web100_log_entry.snap.X_dbg3`	`integer`
`web100_log_entry.snap.X_dbg4`	`integer`
`web100_log_entry.snap.X_rcv_ssthresh`	`integer`
`web100_log_entry.snap.X_wnd_clamp`	`integer`
`web100_log_entry.snap.ZeroRwinRcvd`	`integer`
`web100_log_entry.snap.ZeroRwinSent`	`integer`