2015-08-15
Additional charts
Host zm08.stanford.edu: Total CPU
avg(user) = 1.89 avg(sys) = 0.17 avg(idle) = 95.76 avg(iowait) = 2.15 avg(nice) = 0.00 avg(irq) = 0.00 avg(softirq) = 0.00
Host zm08.stanford.edu: Individual CPU user time
avg(cpu0) = 7.72 avg(cpu1) = 7.96 avg(cpu2) = 2.77 avg(cpu3) = 2.75 avg(cpu4) = 2.16 avg(cpu5) = 2.04 avg(cpu6) = 0.91 avg(cpu7) = 1.22 avg(cpu8) = 0.47 avg(cpu9) = 0.55 avg(cpu10) = 0.38 avg(cpu11) = 0.49 avg(cpu12) = 1.97 avg(cpu13) = 2.06 avg(cpu14) = 2.98 avg(cpu15) = 2.93 avg(cpu16) = 1.60 avg(cpu17) = 1.73 avg(cpu18) = 0.60 avg(cpu19) = 0.70 avg(cpu20) = 0.30 avg(cpu21) = 0.38 avg(cpu22) = 0.24 avg(cpu23) = 0.42
Host zm08.stanford.edu: Individual CPU nice time
avg(cpu0) = 0.00 avg(cpu1) = 0.01 avg(cpu2) = 0.00 avg(cpu3) = 0.00 avg(cpu4) = 0.00 avg(cpu5) = 0.00 avg(cpu6) = 0.00 avg(cpu7) = 0.00 avg(cpu8) = 0.00 avg(cpu9) = 0.00 avg(cpu10) = 0.00 avg(cpu11) = 0.00 avg(cpu12) = 0.00 avg(cpu13) = 0.00 avg(cpu14) = 0.00 avg(cpu15) = 0.00 avg(cpu16) = 0.00 avg(cpu17) = 0.00 avg(cpu18) = 0.00 avg(cpu19) = 0.00 avg(cpu20) = 0.00 avg(cpu21) = 0.00 avg(cpu22) = 0.00 avg(cpu23) = 0.00
Host zm08.stanford.edu: Individual CPU sys time
avg(cpu0) = 0.49 avg(cpu1) = 0.53 avg(cpu2) = 0.20 avg(cpu3) = 0.22 avg(cpu4) = 0.12 avg(cpu5) = 0.15 avg(cpu6) = 0.07 avg(cpu7) = 0.11 avg(cpu8) = 0.05 avg(cpu9) = 0.05 avg(cpu10) = 0.05 avg(cpu11) = 0.05 avg(cpu12) = 0.23 avg(cpu13) = 0.23 avg(cpu14) = 0.47 avg(cpu15) = 0.42 avg(cpu16) = 0.20 avg(cpu17) = 0.20 avg(cpu18) = 0.08 avg(cpu19) = 0.08 avg(cpu20) = 0.07 avg(cpu21) = 0.08 avg(cpu22) = 0.05 avg(cpu23) = 0.07
Host zm08.stanford.edu: Individual CPU idle time
avg(cpu0) = 77.02 avg(cpu1) = 73.22 avg(cpu2) = 96.18 avg(cpu3) = 95.98 avg(cpu4) = 97.18 avg(cpu5) = 97.30 avg(cpu6) = 98.79 avg(cpu7) = 98.43 avg(cpu8) = 99.37 avg(cpu9) = 99.29 avg(cpu10) = 99.50 avg(cpu11) = 99.34 avg(cpu12) = 92.14 avg(cpu13) = 90.81 avg(cpu14) = 95.96 avg(cpu15) = 96.08 avg(cpu16) = 97.84 avg(cpu17) = 97.74 avg(cpu18) = 99.08 avg(cpu19) = 99.04 avg(cpu20) = 99.49 avg(cpu21) = 99.42 avg(cpu22) = 99.63 avg(cpu23) = 99.41
Host zm08.stanford.edu: Individual CPU iowait time
avg(cpu0) = 14.70 avg(cpu1) = 18.21 avg(cpu2) = 0.85 avg(cpu3) = 1.03 avg(cpu4) = 0.53 avg(cpu5) = 0.50 avg(cpu6) = 0.20 avg(cpu7) = 0.23 avg(cpu8) = 0.09 avg(cpu9) = 0.10 avg(cpu10) = 0.07 avg(cpu11) = 0.12 avg(cpu12) = 5.66 avg(cpu13) = 6.89 avg(cpu14) = 0.57 avg(cpu15) = 0.57 avg(cpu16) = 0.35 avg(cpu17) = 0.32 avg(cpu18) = 0.22 avg(cpu19) = 0.16 avg(cpu20) = 0.12 avg(cpu21) = 0.11 avg(cpu22) = 0.07 avg(cpu23) = 0.09
Host zm08.stanford.edu: Individual CPU irq time
avg(cpu0) = 0.00 avg(cpu1) = 0.00 avg(cpu2) = 0.00 avg(cpu3) = 0.00 avg(cpu4) = 0.00 avg(cpu5) = 0.00 avg(cpu6) = 0.00 avg(cpu7) = 0.00 avg(cpu8) = 0.00 avg(cpu9) = 0.00 avg(cpu10) = 0.00 avg(cpu11) = 0.00 avg(cpu12) = 0.00 avg(cpu13) = 0.00 avg(cpu14) = 0.00 avg(cpu15) = 0.00 avg(cpu16) = 0.00 avg(cpu17) = 0.00 avg(cpu18) = 0.00 avg(cpu19) = 0.00 avg(cpu20) = 0.00 avg(cpu21) = 0.00 avg(cpu22) = 0.00 avg(cpu23) = 0.00
Host zm08.stanford.edu: Individual CPU softirq time
avg(cpu0) = 0.07 avg(cpu1) = 0.06 avg(cpu2) = 0.00 avg(cpu3) = 0.00 avg(cpu4) = 0.00 avg(cpu5) = 0.00 avg(cpu6) = 0.00 avg(cpu7) = 0.00 avg(cpu8) = 0.00 avg(cpu9) = 0.00 avg(cpu10) = 0.00 avg(cpu11) = 0.00 avg(cpu12) = 0.00 avg(cpu13) = 0.00 avg(cpu14) = 0.02 avg(cpu15) = 0.00 avg(cpu16) = 0.00 avg(cpu17) = 0.00 avg(cpu18) = 0.00 avg(cpu19) = 0.00 avg(cpu20) = 0.00 avg(cpu21) = 0.00 avg(cpu22) = 0.00 avg(cpu23) = 0.00
Host zm08.stanford.edu: Process CPU
avg(mailbox) = 0.79 avg(mysql) = 0.54 avg(stats) = 0.00
Host zm08.stanford.edu: Process Total Memory
avg(mailbox) = 21167.83 avg(mysql) = 18189.00 avg(stats) = 0.00
Host zm08.stanford.edu: Process Resident Memory
avg(mailbox) = 8702.59 avg(mysql) = 17843.47 avg(stats) = 0.00
Host zm08.stanford.edu: Virtual Memory
avg(swapped) = 43.87 avg(free) = 12080.54 avg(active) = 37925.71 avg(inactive) = 12186.99 avg(page cache) = 13610.23
Host zm08.stanford.edu: Context Switches
avg(context switches) = 1622.97
Host zm08.stanford.edu: Run/Blocked Process Queue Size, Load Average
avg(processes waiting for run time) = 0.94 avg(processes waiting for io time) = 0.89 avg(load average (1 min)) = 1.63
Host zm08.stanford.edu: Disk Utilization (Top 10 avg)
avg(sdf) = 8.35 avg(sdk) = 8.36 avg(dm-0) = 15.43 avg(sdg) = 5.47 avg(sdl) = 5.47 avg(sdh) = 9.12 avg(dm-4) = 11.43 avg(dm-3) = 6.15 avg(sdm) = 9.13 avg(sdu) = 24.17
Host zm08.stanford.edu: Disk Throughput: /dev/sde
avg(sde read) = 0.19 avg(sde write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdf
avg(sdf read) = 275.55 avg(sdf write) = 52.69
Host zm08.stanford.edu: Disk Throughput: /dev/sda
avg(sda read) = 0.19 avg(sda write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdk
avg(sdk read) = 275.44 avg(sdk write) = 52.82
Host zm08.stanford.edu: Disk Throughput: /dev/sdt
avg(sdt read) = 0.19 avg(sdt write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdq
avg(sdq read) = 0.19 avg(sdq write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdj
avg(sdj read) = 0.20 avg(sdj write) = 2.47
Host zm08.stanford.edu: Disk Throughput: /dev/dm-0
avg(dm-0 read) = 550.59 avg(dm-0 write) = 105.51
Host zm08.stanford.edu: Disk Throughput: /dev/sdg
avg(sdg read) = 425.84 avg(sdg write) = 4.92
Host zm08.stanford.edu: Disk Throughput: /dev/dm-1
avg(dm-1 read) = 370.28 avg(dm-1 write) = 502.58
Host zm08.stanford.edu: Disk Throughput: /dev/dm-2
avg(dm-2 read) = 0.00 avg(dm-2 write) = 5.04
Host zm08.stanford.edu: Disk Throughput: /dev/sdl
avg(sdl read) = 425.26 avg(sdl write) = 4.83
Host zm08.stanford.edu: Disk Throughput: /dev/sdc
avg(sdc read) = 0.19 avg(sdc write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdr
avg(sdr read) = 0.19 avg(sdr write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sds
avg(sds read) = 0.19 avg(sds write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdi
avg(sdi read) = 185.23 avg(sdi write) = 251.28
Host zm08.stanford.edu: Disk Throughput: /dev/sdh
avg(sdh read) = 948.99 avg(sdh write) = 0.18
Host zm08.stanford.edu: Disk Throughput: /dev/dm-4
avg(dm-4 read) = 1897.13 avg(dm-4 write) = 0.40
Host zm08.stanford.edu: Disk Throughput: /dev/sdb
avg(sdb read) = 0.19 avg(sdb write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/dm-3
avg(dm-3 read) = 850.70 avg(dm-3 write) = 9.75
Host zm08.stanford.edu: Disk Throughput: /dev/sdm
avg(sdm read) = 948.53 avg(sdm write) = 0.21
Host zm08.stanford.edu: Disk Throughput: /dev/sdp
avg(sdp read) = 0.19 avg(sdp write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdn
avg(sdn read) = 185.44 avg(sdn write) = 251.29
Host zm08.stanford.edu: Disk Throughput: /dev/sdu
avg(sdu read) = 23.56 avg(sdu write) = 35.91
Host zm08.stanford.edu: Disk Throughput: /dev/sdd
avg(sdd read) = 0.19 avg(sdd write) = 0.00
Host zm08.stanford.edu: Disk Throughput: /dev/sdo
avg(sdo read) = 0.20 avg(sdo write) = 2.57
Host zm08.stanford.edu: Disk IOPs: /dev/sde
avg(sde read) = 0.05 avg(sde write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdf
avg(sdf read) = 17.68 avg(sdf write) = 3.54
Host zm08.stanford.edu: Disk IOPs: /dev/sda
avg(sda read) = 0.05 avg(sda write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdk
avg(sdk read) = 17.68 avg(sdk write) = 3.54
Host zm08.stanford.edu: Disk IOPs: /dev/sdt
avg(sdt read) = 0.05 avg(sdt write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdq
avg(sdq read) = 0.05 avg(sdq write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdj
avg(sdj read) = 0.05 avg(sdj write) = 0.14
Host zm08.stanford.edu: Disk IOPs: /dev/dm-0
avg(dm-0 read) = 35.26 avg(dm-0 write) = 7.08
Host zm08.stanford.edu: Disk IOPs: /dev/sdg
avg(sdg read) = 24.41 avg(sdg write) = 0.60
Host zm08.stanford.edu: Disk IOPs: /dev/dm-1
avg(dm-1 read) = 9.91 avg(dm-1 write) = 2.33
Host zm08.stanford.edu: Disk IOPs: /dev/dm-2
avg(dm-2 read) = 0.00 avg(dm-2 write) = 0.28
Host zm08.stanford.edu: Disk IOPs: /dev/sdl
avg(sdl read) = 24.41 avg(sdl write) = 0.60
Host zm08.stanford.edu: Disk IOPs: /dev/sdc
avg(sdc read) = 0.05 avg(sdc write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdr
avg(sdr read) = 0.05 avg(sdr write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sds
avg(sds read) = 0.05 avg(sds write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdi
avg(sdi read) = 5.00 avg(sdi write) = 1.16
Host zm08.stanford.edu: Disk IOPs: /dev/sdh
avg(sdh read) = 19.46 avg(sdh write) = 0.02
Host zm08.stanford.edu: Disk IOPs: /dev/dm-4
avg(dm-4 read) = 38.83 avg(dm-4 write) = 0.03
Host zm08.stanford.edu: Disk IOPs: /dev/sdb
avg(sdb read) = 0.05 avg(sdb write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/dm-3
avg(dm-3 read) = 48.72 avg(dm-3 write) = 1.20
Host zm08.stanford.edu: Disk IOPs: /dev/sdm
avg(sdm read) = 19.46 avg(sdm write) = 0.02
Host zm08.stanford.edu: Disk IOPs: /dev/sdp
avg(sdp read) = 0.05 avg(sdp write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdn
avg(sdn read) = 5.00 avg(sdn write) = 1.16
Host zm08.stanford.edu: Disk IOPs: /dev/sdu
avg(sdu read) = 0.85 avg(sdu write) = 2.07
Host zm08.stanford.edu: Disk IOPs: /dev/sdd
avg(sdd read) = 0.05 avg(sdd write) = 0.00
Host zm08.stanford.edu: Disk IOPs: /dev/sdo
avg(sdo read) = 0.05 avg(sdo write) = 0.14
Host zm08.stanford.edu: Disk Usage
avg(/opt/zimbra/redo-mysql) = 2.00 avg(/usr/vice/cache) = 12.95 avg(/opt/zimbra/store) = 92.00 avg(/opt/zimbra) = 76.00 avg(/opt/zimbra/hsm1) = 96.00 avg(/) = 8.01 avg(/opt/zimbra/redolog) = 10.69
Host zm08.stanford.edu: Open file descriptor count
avg(total system file descriptors open) = 4816.50 avg(mailboxd file descriptors open) = 1411.09
Host zm08.stanford.edu: Process CPU Time (Top 10 avg)
avg(tripwire) = 2123.00 avg(updatedb) = 41.81 avg(java) = 975.19 avg(gzip) = 2814.29 avg(mysqld) = 401.26 avg(httpd) = 8.46 avg(mysqldump) = 136.19 avg(zmmailboxd) = 584.02 avg(service) = 6.50 avg(zmstat-allprocs) = 11.30
Host zm08.stanford.edu: SOAP: Invocation Count: Summary (Top 10 max)
max(GetMsgRequest) = 25.00 max(ConvActionRequest.read) = 29.00 max(SearchConvRequest) = 30.00 max(AutoCompleteRequest) = 14.00 max(SearchRequest) = 78.00 max(ConvActionRequest.delete) = 9.00 max(SyncRequest) = 23.00 max(ConvActionRequest.trash) = 31.00 max(WaitSetRequest) = 23.00 max(GetDomainInfoRequest) = 40.00
Host zm08.stanford.edu: SOAP: Average Call Duration: Summary (Top 10 avg)
avg(SearchRequest) = 6520.70 avg(SendMsgRequest) = 440.68 avg(DeleteAccountRequest) = 40355.00 avg(MsgActionRequest.delete) = 377.00 avg(CreateMountpointRequest) = 94.67 avg(BackupQueryRequest) = 270.83 avg(GetAggregateQuotaUsageOnServerRequest) = 901.00 avg(ConvActionRequest.trash) = 105.75 avg(WaitSetRequest) = 80.99 avg(GetQuotaUsageRequest) = 1024.00
Host zm08.stanford.edu: MySQL: InnoDB Buffer Pool Pages
avg(pages read) = 503245563.88 avg(pages written) = 178778385.86
Host zm08.stanford.edu: MySQL: InnoDB Buffer Pool Hit Rate
avg(buffer pool hit rate) = 992.67
Host zm08.stanford.edu: MySQL: Database Connections In Use
avg(connections in use) = 0.18
Host zm08.stanford.edu: MySQL: Tables Open/Opened
avg(open_tables) = 1167.81 avg(opened_tables (cumulative)) = 532949.76
Host zm08.stanford.edu: MySQL: Total Slow Queries Count
avg(slow queries (cumulative)) = 742118.80
Host zm08.stanford.edu: Mailboxd: Connection Pool Get Latency
avg(mysql) = 0.25 avg(ldap) = 0.02
Host zm08.stanford.edu: Mailboxd: Lucene IO
avg(Index writes) = 4.01 avg(Index reads) = 10.52
Host zm08.stanford.edu: Mailboxd: LMTP Delivery Throughput
avg(received) = 1.61 avg(delivered) = 1.62
Host zm08.stanford.edu: Mailboxd: LMTP Delivery Rate
avg(received msgs) = 0.01 avg(delivered msgs) = 0.01 avg(received x rcpts) = 0.01
Host zm08.stanford.edu: Mailboxd: Mailbox Add Rate (Delivery Rate)
avg(mailbox add rate) = 0.01
Host zm08.stanford.edu: Mailboxd: Mailbox Add Latency (Delivery Speed)
avg(mailbox add latency) = 30.65
Host zm08.stanford.edu: Mailboxd: Request Rate by Client Protocol
avg(SOAP) = 0.10 avg(IMAP) = 0.40 avg(POP) = 0.07
Host zm08.stanford.edu: Mailboxd: Response Time by Client Protocol
avg(SOAP) = 1263.35 avg(IMAP) = 12.98 avg(POP) = 15.88
Host zm08.stanford.edu: Mailboxd: Active Connections by Client Protocol
avg(IMAP) = 2.82 avg(IMAP SSL) = 6.32 avg(POP) = 0.00 avg(POP SSL) = 0.04 avg(SOAP sessions) = 2.39
Host zm08.stanford.edu: Mailboxd: Mailbox Get Count
avg(mailbox get count) = 37.00
Host zm08.stanford.edu: Mailboxd: Mailbox Get Latency
avg(mailbox get latency) = 0.05
Host zm08.stanford.edu: Mailboxd: Mailbox Cache Hit Rate
avg(mailbox cache hit rate) = 98.15
Host zm08.stanford.edu: Mailboxd: Mailbox Item/Blob Cache Hit Rate
avg(msg blob cache) = 20.76 avg(item metadata cache) = 45.16
Host zm08.stanford.edu: Mailboxd: Minor Garbage Collection Time
avg(minor GC time) = 0.03
Host zm08.stanford.edu: Mailboxd: Major Garbage Collection Time
avg(major GC time) = 0.00
Host zm08.stanford.edu: Mailboxd: Minor Garbage Collection Count
avg(minor GC count) = 0.87
Host zm08.stanford.edu: Mailboxd: Major Garbage Collection Count
avg(major GC count) = 0.00
Host zm08.stanford.edu: Mailboxd: JVM Heap Used
avg(total) = 4716.78 avg(PS Old Gen) = 0.00 avg(CMS Old Gen) = 3824.65 avg(Tenured Gen) = 0.00 avg(Train Gen) = 0.00 avg(PS Eden Space) = 0.00 avg(PS Survivor Space) = 0.00 avg(Par Eden Space) = 811.12 avg(Par Survivor Space) = 81.01 avg(Eden Space) = 0.00 avg(Survivor Space) = 0.00
Host zm08.stanford.edu: Mailboxd: JVM Heap Free
avg(total free) = 3110.53 avg(PS Old Gen free) = 0.00 avg(CMS Old Gen free) = 2196.35 avg(Tenured Gen free) = 0.00 avg(Train Gen free) = 0.00 avg(PS Eden Space free) = 0.00 avg(PS Survivor Space free) = 0.00 avg(Par Eden Space free) = 794.51 avg(Par Survivor Space free) = 119.68 avg(Eden Space free) = 0.00 avg(Survivor Space free) = 0.00
Host zm08.stanford.edu: Mailboxd: JVM Permanent Generation and Code Cache
avg(PS Perm Gen) = 0.00 avg(Perm Gen) = 0.00 avg(CMS Perm Gen) = 127.55 avg(Code Cache) = 39.35 avg(Perm Gen [shared-ro]) = 0.00 avg(Perm Gen [shared-rw]) = 0.00 avg(PS Perm Gen free) = 0.00 avg(Perm Gen free) = 0.00 avg(CMS Perm Gen free) = 82.45 avg(Code Cache free) = 0.34 avg(Perm Gen [shared-ro] free) = 0.00 avg(Perm Gen [shared-rw] free) = 0.00
Host zm08.stanford.edu: convertd: CPU time used
avg(total cputime) = 0.07 avg(user time) = 0.06 avg(system time) = 0.00
Host zm08.stanford.edu: convertd: Resident Memory
avg(convertd rss) = 170.57
Host zm08.stanford.edu: convertd: Processes and Threads
avg(processes) = 5.41 avg(threads) = 5.41
No data available for the following charts:
Host zm08.stanford.edu: Swap Activity
Host zm08.stanford.edu: Disk Throughput: /dev/dm-7
Host zm08.stanford.edu: Disk Throughput: /dev/sdx
Host zm08.stanford.edu: Disk Throughput: /dev/sdab
Host zm08.stanford.edu: Disk Throughput: /dev/sdaa
Host zm08.stanford.edu: Disk Throughput: /dev/sdad
Host zm08.stanford.edu: Disk Throughput: /dev/sdag
Host zm08.stanford.edu: Disk Throughput: /dev/sdaf
Host zm08.stanford.edu: Disk Throughput: /dev/sdy
Host zm08.stanford.edu: Disk Throughput: /dev/sdv
Host zm08.stanford.edu: Disk Throughput: /dev/sdac
Host zm08.stanford.edu: Disk Throughput: /dev/dm-5
Host zm08.stanford.edu: Disk Throughput: /dev/sdw
Host zm08.stanford.edu: Disk Throughput: /dev/sdae
Host zm08.stanford.edu: Disk Throughput: /dev/dm-6
Host zm08.stanford.edu: Disk Throughput: /dev/sdz
Host zm08.stanford.edu: Disk IOPs: /dev/dm-7
Host zm08.stanford.edu: Disk IOPs: /dev/sdx
Host zm08.stanford.edu: Disk IOPs: /dev/sdab
Host zm08.stanford.edu: Disk IOPs: /dev/sdaa
Host zm08.stanford.edu: Disk IOPs: /dev/sdad
Host zm08.stanford.edu: Disk IOPs: /dev/sdag
Host zm08.stanford.edu: Disk IOPs: /dev/sdaf
Host zm08.stanford.edu: Disk IOPs: /dev/sdy
Host zm08.stanford.edu: Disk IOPs: /dev/sdv
Host zm08.stanford.edu: Disk IOPs: /dev/sdac
Host zm08.stanford.edu: Disk IOPs: /dev/dm-5
Host zm08.stanford.edu: Disk IOPs: /dev/sdw
Host zm08.stanford.edu: Disk IOPs: /dev/sdae
Host zm08.stanford.edu: Disk IOPs: /dev/dm-6
Host zm08.stanford.edu: Disk IOPs: /dev/sdz
Host zm08.stanford.edu: Mailboxd: Dirty Lucene Index Writers
Host zm08.stanford.edu: Mailboxd: Lucene IndexWriterCache Hitrate
Host zm08.stanford.edu: nginx: CPU time used
Host zm08.stanford.edu: nginx: Resident Memory
Host zm08.stanford.edu: nginx: Processes and Threads