# DirectRandomAccessReader.read() and seek() generate performance bottleneck

**URL:** <https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086>\
**Category:** Wowza Streaming Engine\
**Created:** [June 17, 2014, 9:24am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086 "2014-06-17T09:24:29Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 17, 2014, 9:24am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/1 "2014-06-17T09:24:29Z")

</div>

Hi,

I’m currently tuning performance of wowza 4.0.3, with following hardware and test situation, we can get about 8Gbps-9Gbps.

- 2\*Intel® Xeon® CPU E5-2690 0 @ 2.90GHz(total 32 logic processors)

- 32G RAM

- Only one vod file, H.264 encoded, 5Mbps, 1.57GBytes, located in {WowzaInstallDir}/content, which is a tmpfs directory by command “mount -t tmpfs -o size=2048m tmpfs {WowzaInstallDir}/content”

- Clients use HLS to access VOD video

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [June 17, 2014, 1:59pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/2 "2014-06-17T13:59:05Z")

</div>

Hi,

How are your test clients connecting to the stream? Are they connecting via the network or via localhost? What protocol are they using?

The 5mbps file x 2000 connections is 10gbps so is it possible you are reaching the limits of the network connection (assuming a 10gbps network)?

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [June 17, 2014, 4:14pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/3 "2014-06-17T16:14:40Z")

</div>

Hi,

Thanks for the info.

Can you please let us know how you are generating the connections. It would be interesting to replicate your configuration in order to perform some tests.

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [June 17, 2014, 9:54pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/4 "2014-06-17T21:54:13Z")

</div>

Hi,

The read() & seek() methods basically wrap the same methods in the Java RandomAccessFile.

I’ll run some tests tomorrow to see if we can see anything.

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [June 19, 2014, 4:30pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/5 "2014-06-19T16:30:15Z")

</div>

Hi,

I’m not sure if there is an issue with wowza or with your script. I can start around 2500 connections with no issues but then when I start more, something starts going wrong and there are 1000’s of connections started at the same time.

Looking at the script, when the thread hits the end of the while loop, either due to an error or a stream end, it restarts immediately without sleeping. At this stage, an error is thrown again immediately and it loops again quickly. I’m not sure if the script is then making a bad request or it is just trying to request the urls too quickly but when it happens, Wowza uses up all of it’s max allowed files very quickly. I think you may need to put the thread to sleep for a bit before retrying or shut down that thread and create a new one.

Can you see if you can get more logging for when the stream errors occurs. It may give us a clue as to what the server has sent that the script thinks is an error.

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [June 19, 2014, 5:01pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/6 "2014-06-19T17:01:03Z")

</div>

I had a look at the errors being thrown and it may be possible the server is running out of TCP sockets. This is the error. Thread 39 the exception is: \<urlopen error [Errno 99] Cannot assign requested address\>

Doing a netstat, I see nearly 30k open sockets in TIME\_WAIT.

Normally, HTTP streaming will reuse the same socket for a number of requests. It looks like your script is creating a new socket for each request which could cause issues.

Update: Setting **net.ipv4.tcp\_tw\_recycle = 1** allows the sockets to be reused quickly. With this set, I can get around 5000 connections. Over that, I am running out of cpu. Please try that setting with your tests. I also added a sleep to the end of the thread loop.

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [June 20, 2014, 2:47pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/7 "2014-06-20T14:47:07Z")

</div>

Hi,

I was just using a single file and it was around 1mbps. I was hitting cpu limits with only 8 cores available.

Also look at the thread pool sizes. Increasing these from the default settings may help, especially as you have 32 cores.

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [June 23, 2014, 5:31pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/8 "2014-06-23T17:31:14Z")

</div>

Hi,

I will try with a larger file.

I’m not sure what you would gain by implementing your own reader as the read & seek methods in the DirectRandomAccessReader class are about as close as you can get to the java RandomAccessFile methods that they call.

Saying that, you can create you own reader class. You would have to implement IRandomAccessReader & ITrackRandomAccessReaderPerformance and handle all their methods. See [https://www.wowza.com/docs/media-cache-implementation-that-will-first-try-to-access-content-locally-before-getting-it-from-a-remote-source-mediacachelocalcontent](https://www.wowza.com/docs/media-cache-implementation-that-will-first-try-to-access-content-locally-before-getting-it-from-a-remote-source-mediacachelocalcontent) as an example of how to create a custom reader class. The source code is available in the [Module Collection](https://www.wowza.com/docs/utility-modules-for-wowza-streaming-engine-media-server-software) download.

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [July 4, 2014, 8:41am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/9 "2014-07-04T08:41:24Z")

</div>

The OS parameters can be set with the sysctl command and made persistent with the /etc/sysctl.conf file.

The Wowza parameters you mention can be set via the Wowza Streaming Engine Manager UI in the Server \> Performance Tuning section.

Roger.

---

<div class="post-metadata">

**Author:** ![Roger\_Littin1](https://avatars.discourse-cdn.com/v4/letter/r/c89c15/32.png) [@Roger\_Littin1](https://community.wowza.com/u/Roger_Littin1)\
**Post date:** [July 4, 2014, 7:44pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/10 "2014-07-04T19:44:48Z")

</div>

Hi,

The only setting I have changed from standard is the net.ipv4.tcp\_tw\_reuse to enable it. The Wowza settings are default production settings.

Please note, I have really only needed to change the above setting to allow your python script to work properly. You shouldn’t need to do so normally as the real connections wouldn’t be coming from localhost or all from the same machine.

Roger.

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 17, 2014, 11:40am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/11 "2014-06-17T11:40:41Z")

</div>

> Hi,
> 
> How are your test clients connecting to the stream? Are they connecting via the network or via localhost? What protocol are they using?
> 
> The 5mbps file x 2000 connections is 10gbps so is it possible you are reaching the limits of the network connection (assuming a 10gbps network)?
> 
> Roger.

Thanks, we use localhost, so network should not be the limits. And we can see that, when using 2000 clients, the Bytes Out speed goes down to about 6Gbps-7Gbps

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 17, 2014, 1:29pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/12 "2014-06-17T13:29:49Z")

</div>

> Hi,
> 
> Thanks for the info.
> 
> Can you please let us know how you are generating the connections. It would be interesting to replicate your configuration in order to perform some tests.
> 
> Roger.

Thanks for rapid response.

We use python script as as simple client, its content as follows. To start 200 client connection, using command “python wowzatest2.py 200 &”, where ‘wowzatest2.py’ is the name of python script file, and if you want to start 2000 connection, execute above command 10 times to avoid performance limit of python itself.

Could you provide some details about DirectRandomAccessReader.read() and seek() ? It look like they are native method?

```auto
import httplib
import urllib2
import random
import time
import sys
import logging
import thread 
import getopt
urlString = "http://10.0.63.180:1935/vod/"
id_range_start = 1
id_range_end = 1
ERRORNUM = 0
global CONSOLE
def initLogging(logFilename = None):
    """Init for logging
    """
    if logFilename is None:
        logFilename = "run.log"
    logging.basicConfig(    
                    level = logging.DEBUG,
                    format = 'LINE %(lineno)-4d %(levelname)-8s %(message)s',
                    datefmt = '%m-%d %H:%M',
                    filename = logFilename,
                    filemode = 'w');
    # define a Handler which writes INFO messages or higher to the sys.stderr
    #CONSOLE = logging.StreamHandler();
    #CONSOLE.setLevel(logging.INFO);
    # set a format which is simpler for console use
    #formatter = logging.Formatter('LINE %(lineno)-4d : %(levelname)-8s %(message)s');
    # tell the handler to use this format
    #CONSOLE.setFormatter(formatter);
    #logging.getLogger().addHandler(CONSOLE);
#-------------------------------------------------------------------------------  
def destoryLogging():
    logging.getLogger().removeHandler(CONSOLE)
def get_url_from_wowza_m3u8(url,vod_id,threadNo):
    try:
        rawData = urllib2.urlopen(url).readlines()
        for data in rawData:
            if(('.m3u8') in data):
                playm3u8 = urlString + "mp4:POI_" + vod_id + ".mp4/" + data 
        logging.info("Thread "+str(threadNo)+"get_url_from_wowza_m3u8() return: " + str(playm3u8))
	return playm3u8
    except Exception, e:
        logging.error("Thread "+str(threadNo)+"Error! get_url_from_wowza_m3u8() via url: " + str(url))
        logging.error("Thread "+str(threadNo)+"the exception is: " + str(e))
        errorData = ['error']
        return errorData
def get_ts(playm3u8,vod_id,threadNo):
    try:
        rawData = urllib2.urlopen(playm3u8).readlines()
        #logging.info("----------m3u8 content: " + str(rawData))
        for data in rawData:	
            if(('EXTINF') in data):        
                    tsDuration=int(round(float(data[8:12])))
                    logging.info("duration: " + str(tsDuration))
            if(('.ts') in data):
                    tsurl = urlString + "mp4:POI_" + vod_id + ".mp4/" + data;
                    beginTime = time.time()
                    #logging.info("----------start time: " + str(beginTime))
                    get_ts_from_url(tsurl)
                    endTime = time.time()
                    #logging.info("----------end time: " + str(endTime))
                    logging.info("Thread "+str(threadNo)+" get_ts:ok")
		   # logging.info("end-start:"+str(int(endTime-beginTime)))
		    #logging.info("tsDuration:"+str(tsDuration))
                    sleep_period = tsDuration - int(endTime - beginTime)
                    logging.info("sleep " + str(sleep_period))
                    if (sleep_period >= 0):
                        time.sleep(sleep_period)
                    else:
                        logging.error("Thread "+str(threadNo)+" get ts spend "+str(endTime - beginTime)+",that's too long")
    except Exception, e:
        logging.error("Thread "+str(threadNo)+" get_ts() via url: " + str(playm3u8))
        logging.error("Thread "+str(threadNo)+" the exception is: " + str(e))
        
def url_of_wowaz(urlString,vod_id):
    url = urlString + "mp4:POI_" + vod_id + ".mp4/playlist.m3u8"
    return url
def get_ts_from_url(ts_url):
    try:
        tsData = urllib2.urlopen(ts_url).read()
# ts_file_name = str(time.time())
# f_ts = file(ts_file_name + '.ts','wb')
# f_ts.write(tsData)
# f_ts.close()
        return "ok"
    except Exception, e:
        logging.error("Thread "+str(threadNo)+" get_ts_from_url() via url: " + str(ts_url))
        logging.error("Thread "+str(threadNo)+" the exception is: " + str(e))
        return "error"
def test(no):
    randomtime=random.randint(0,13)
    time.sleep(randomtime)
    while(1):
        logging.info(str(no)+" start:")
        vod_id = str(random.randint(id_range_start, id_range_end))
        playlist_m3u8 = url_of_wowaz(urlString,vod_id)
        logging.info(str(no)+" the playlist_m3u8 is " + str(playlist_m3u8))
        ts_m3u8 = get_url_from_wowza_m3u8(playlist_m3u8,vod_id,no)
        logging.info(str(no)+"---------- the ts_m3u8 is " + str(ts_m3u8))
        get_ts(ts_m3u8,vod_id,no)
        logging.info(str(no)+"---------- get_all_ts:ok")
        logging.info(str(no)+"---- end:")
if __name__ ==' __main__':
    logname = str(time.time())+ ".log"
    initLogging(logname)
    for i in range(1,int(sys.argv[1])):
    	print "start thread ",i
    	thread.start_new_thread(test,(i,)) 
    test(0)

```

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 17, 2014, 4:40pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/13 "2014-06-17T16:40:46Z")

</div>

Hi, Roger, could you give me some suggests? Thanks.

Or could you give some detailed information about DirectRandomAccessReader.read() and seek(), for example, the java code? Em, I’m a software engineer, I like source code :), Thank you very much.

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 18, 2014, 9:12am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/14 "2014-06-18T09:12:28Z")

</div>

Hi,

Today, I copy the VOD file as anther VOD file, and clients access them randomly, then the max client connections increase to 2400. Em, It looks like some limit of the file access, may be file-lock?

DaLi

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 20, 2014, 10:40am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/15 "2014-06-20T10:40:03Z")

</div>

En, Thanks, I will set net.ipv4.tcp\_tw\_recycle = 1 and modify the client script, then perform a test.

How many Vod files do you use? And what’s the bitrate, size, and frame resolution of them?

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 20, 2014, 11:36am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/16 "2014-06-20T11:36:40Z")

</div>

What you suggest about thread pool size? At first, I user the default 600/400, the max client connections number is 1400, and then I change it to 64/32, the max client connections number increase to 1800.

Wait a moment please, I will post the test result with net.ipv4.tcp\_tw\_reuse = 1

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 20, 2014, 11:49am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/17 "2014-06-20T11:49:22Z")

</div>

Em, net.ipv4.tcp\_tw\_reuse = 1 took effect, there is almost no TIME\_WAIT connection, but the max support client connections did not changed, it remained about 1800.

Following is the netstat’s output (12455 is wowza engine’s PID):

[root@boss ~]# netstat -nap |grep 12455 | grep TIME\_WAIT | wc -l

0

[root@boss ~]# netstat -nap |grep 12455 | grep ESTABLISHED | wc -l

1721

[root@boss ~]# netstat -nap |grep 12455 | wc -l

1781

And none error logs appeared in wowza’s logs.

The VisualVM’s output of “thread” tag is as following, green means the thread’s state is running.

![](http://img.blog.csdn.net/20140620115439843?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbGlkYWJudQ==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center)

It looks like that performance bottleneck of my test case is different from yours. Could you test a vod file with 5Mbps, 1280\*720 frame resolution? Thanks.

And I transcode my testing vod file to 1Mbps, test again, which can reach max 2400 client connection, and then the above image appears.

Could you give me your parameters including OS’s tune parameters and Wowza’s parameters? Thanks

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [June 23, 2014, 11:54am UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/18 "2014-06-23T11:54:52Z")

</div>

Hello, Roger, are you tracking this issue? Thanks.

Another question, can I using a extended class instead of DirectRandomAccessReader? I would like to override its read and seek.

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [July 3, 2014, 1:09pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/19 "2014-07-03T13:09:27Z")

</div>

Hello, Roger

Could you tell me how do you set the server’s parameters? Including OS’s (net.ipv4.tcp\_tw\_reuse, net.core.rmem\_max, etc) and Wowza’s (GC method, ServerTransportThread Number,etc)

Thanks.

---

<div class="post-metadata">

**Author:** ![Da\_Li](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@Da\_Li](https://community.wowza.com/u/Da_Li)\
**Post date:** [July 4, 2014, 1:50pm UTC](https://community.wowza.com/t/directrandomaccessreader-read-and-seek-generate-performance-bottleneck/43086/20 "2014-07-04T13:50:11Z")

</div>

> The OS parameters can be set with the sysctl command and made persistent with the /etc/sysctl.conf file.
> 
> The Wowza parameters you mention can be set via the Wowza Streaming Engine Manager UI in the Server \> Performance Tuning section.
> 
> Roger.

Em, I did not say clearly.

I mean which system parameters you have changed, and changed to what.

could you execute command “sysctl -p”, " cat /proc/cpuinfo" on your testing server and paste the output here?

Thanks.