16 Reply by Phalanger

  • Phalanger
  • Member
  • Offline

I'm have created a DirectX style control, which uses Direct2D right now (through SlimDX).  I can draw a rectangle in it and fill it, so next step is to fill it with the bitmap data and ensure the right ratio.  Then you would have basic output (no items drawn on it, but it's the starts).

Just being Direct2D it would only work on Vista/7 (I don't believe any of the XP updates included it).  The container also support DirectX9/10, however I am not attempting them at the moment as my knowledge is not that good.  Just took me a little while to get use to C# (I'm from the C++ world wink).

17 Reply by Phalanger

  • Phalanger
  • Member
  • Offline

I have it drawing the image, but as soon as it appears, it goes blank again.  This is not happening in the test projects.  Is there something in the code which would be rendering this?

18 Reply by Phalanger (edited by Phalanger 2010-11-06 13:07:22)

  • Phalanger
  • Member
  • Offline

Ok, rendering on an external form seems good, with smooth playback with HD videos which before were issues.  However it does jump now and then (it's not clean the code as it was a hack around).

One thing which I think must be changed is the use of the invoking the paint method on the picture box.  This is not a fast method.  It would be much better to call this function directly (and maybe from an external loop).  Going through the operating system/control seems like a possible performance issue.

http://blogs.msdn.com/b/tmiller/archive … 15008.aspx
http://slimdx.mdxinfo.com/tutorials/basicwindow.php

I have a feeling the only way it will really work well (like high performance is rewriting this control).  That way frames are extracted straight into the working canvas memory, and drawn from there.  Other drawings on top are added to a list, which adds them to the rendering line to place on top of the image as it is renders.  Maybe when you have more than one video, they should be rendered on the same surface (even if they have separate controls.

Does this sound right?

19 Reply by joan

  • joan
  • joan
  • Admin
  • Offline
Phalanger wrote:

Ok, rendering on an external form seems good, with smooth playback with HD videos which before were issues.  However it does jump now and then (it's not clean the code as it was a hack around).

It sounds super promising! big_smile
It would be nice to have some instrumentation on this, monitoring how many ms are spent on the rendering when using DX and when using GDI+.

Phalanger wrote:

One thing which I think must be changed is the use of the invoking the paint method on the picture box.  This is not a fast method.  It would be much better to call this function directly (and maybe from an external loop).  Going through the operating system/control seems like a possible performance issue.

Yes, you're probably right. I don't quite remember the rationale for always going through the system's paint event. It's possible that it was simply the most convenient way to get a reference on the Graphic object we are painting on.
The timer itself runs in another thread, so asking for the next frame is done asynchronously and I don't think this can change. However, when we have the frame in question, rendering it on the screen can (should) be triggerred right away. As you point, no need to go through the Windows message queue, that can only slow things down.

Phalanger wrote:

I have a feeling the only way it will really work well (like high performance is rewriting this control).  That way frames are extracted straight into the working canvas memory, and drawn from there.  Other drawings on top are added to a list, which adds them to the rendering line to place on top of the image as it is renders.  Maybe when you have more than one video, they should be rendered on the same surface (even if they have separate controls.

Does this sound right?

I'm not sure I followed everything, if I understood correctly you suggest extracting the frames from FFMPeg directly onto the canvas / graphic card memory, instead of using an intermediate Bitmap object. (?).

I don't know how this would work with the working zone concept. Currently the user may apply some image filters like "Auto Contrast" or "Sharpen" for example. These filters are applied on pre-extracted copies of the frames, to speed things up and add flexibility (combining filters, etc.).
I'm not sure we would be able to make the whole thing fast enough to allow for these filters to be applied in real time and still play smoothly…

The working zone and its pre-extraction of frames helps in many other regards too. For example, you can jump straight to any frame without having to decode all the ones in between (this wouldn't be quite possible without pre-extraction, we would have to seek to the closest keyframe (in the video coding sense) and then start decoding from there).
Also, some effects will always need to have a list of frames in memory anyway, e.g. the "overview" and "reverse" functions, and in the future the more complex special effects like chronophotography, etc. So far this model has proven very convenient.

20 Reply by Phalanger (edited by Phalanger 2010-11-07 01:32:12)

  • Phalanger
  • Member
  • Offline

I'm thinking that there should be two new threads added.
One should be for the decoding, and it places the images into a buffer.  So like 5/10 frames ahead of time.

Then when the timer is called the display classes can take the image and do their rendering on it (like DX, GDI+ based classes which run their own threads).  The timer would also induce the decoding thread to check the buffer.  This would mean moving the drawing code out of the ui and into separate classes.

This would mean that items drawn onto of the image would also have to be put into lists to be drawn by the rendering classes.

Hopefully it would mean workload can be split a bit across the hardware and that any slow down by the OS/UI/decoding does not effect the rendering process.

21 Reply by Phalanger (edited by Phalanger 2010-11-07 01:30:54)

  • Phalanger
  • Member
  • Offline

Currently I can get the image painted onto the centre panel without any issues like this picture box.

I get 25fps on a high resolution video which I use to get around 12.  However some high quality videos (full HD: 720p/1080) I'm still struggling around 13 (but they were worse before).  I find they render smooth then jump, so I feel this is hanging up in other regions, as the ui also hangs.

I think the best was it to make a clean system like above (unless you can think of another way).

22 Reply by joan

  • joan
  • joan
  • Admin
  • Offline

I've started to read about Direct2D and I must say it's pretty exiting. smile
Moving the rendering to the graphic card will not only increase performance through hardware acceleration, it will also free the CPU for other computations.
Stuff like tracking multiple points might be able to run in real time, that would be awesome.

The drawback is portability to other OS (although I've seen something about X support…) and to older Windows versions. (Which might mean having to maintain/release two versions, but this will be a good excuse to finally try some of the new features of .NET 3.5, like the graphic tablet API).

Anyway, yes, it would be good to use this opportunity to refactor the player screen and decoding/rendering flow.
I have added a diagram of the current architecture here to help.

I like the idea of bufferring the incoming images. It won't be necessary when images are already extracted to memory though, but when decoding directly from file, it will help smoothing out the decoding time.

On the other side of the flow, when outputting to screen, the ideal way is like you say independant modules that take the image and data and draw them on their surface, be it GDI+, Direct2D or whatever.
(An "output modules" architecture would make more sense when we want to add back support for document rendering, we would just have to implement a PDF or ODF renderer.)

Passing the list of items to the renderer would completely change the approach for drawing though… As of now, each drawing has a Draw method, and is responsible for drawing itself on the GDI+ surface according to its own internal data. It's convenient and good encapsulation.

Abstracting the output would mean completely redesign this aspect. This needs to be thought carefully.
For instance, the exact same flow is used when saving snapshot or video. We just pass a new Bitmap to draw on instead of the control's Graphic.

One other way would be if renderers could be abstracted and the concrete selected renderer passed to the Draw method of each object.
The object would draw themselves using universal primitives, and these would be translated to actual methods in the concrete renderer.

What do you think ?

23 Reply by Phalanger

  • Phalanger
  • Member
  • Offline

I noticed the drawing section however I think there will be two main issues with it.

Firstly Direct2D, GDI+ and DirectX have totally different drawing methods and it would be best to let them draw it in their own special ways.  This would give the best speed boost (and quality).  Possible file outputs would be able to tag onto the GDI system for many things I guess (unless you would rather use the file types' features instead of bitmaps).

Also this would allow the control to be totally separate of the drawing process which means they can operate by themselves.



I think this is the overall structure that I'm looking towards:

Part A:

Clean PlayerScreenUserInterface2 (maybe lets start with version 3?).
- Remove all drawing code
- Make it contain only the controls/key frames
- Make it responsible for setting values to be used by rendering/decoding etc.


Part B:

Create a new form that is responsible for video output.
- This allows for it to have it's own message handler (useful for cleaning up direct2D/X class on closing which other controls have trouble doing).
- Minimum drawing requirements
- Only one in the application (so multiple videos are rendered on the same form, even where there is more than one PlayerScreenUserInterface).  This should allow for easily adding more videos, blending etc., and not need the PlayerScreenUserInterface to be aware of each other.
- Maybe it could also come to a point where you could position, size the videos over each other with dragging.
- Single GDI+ print, Direct2D/X process for the whole application (not two/3/4 separate ones).


Create a buffer class
- Should be one for each video
- Input/output system.
- Responds to the decoder if more input is require.
- Responds to the drawing system if new image to draw.
- Fills up when opening a video (maybe by calling the decoder which checks if input required, then keeps filling it up till no more input to this class is required).
- Based on some style on linked list system (first in, first out).  I don't know if C# has a type for this.


Create a drawing data class
- Should be one for each video
- Stores all objects to be drawn on the video image (as data, not bitmaps).
- PlayerScreenUserInterface will add the objects to be draw, remove them or clear them (as the user interacts).
- Render will call for a list of the objects to be drawn, and render them onto its video image which the class belongs.


Part C:

Clean timer:
- Should run in its own thread (understand it does).
- On call will tell the drawing class to render images/data to screen.
- On call will tell the decoding class to check the buffer and fill it up.

Create a decoding class:
- Runs in its own thread.
- Checks the buffer class, and if told it needs to be filled up, will place an image into the buffer class (until it knows the class is full or the next timer tick).
- Called by timer, or by video loading.

Create a drawing class:
- Runs in its own thread.
- Checks the buffers have a new image to output.
- Passes the images and drawing data to the output class (like Direct2D/X, GDI+), including where on the screen they should be draws.
- This means it can tell images to be drawn next to each other, over top of each other etc.

Create the output classes:
- Called by the drawing class.
- Set-up the output if not already.
- Draws the images in their locations and draws objects on them using the drawing data.

24 Reply by Phalanger

  • Phalanger
  • Member
  • Offline

Just thinking that the data to draw something on the screen could actually be objects (derived from a base class).  Then the output classes can run through each object, checking its type and then sending them to the function to draw that type.

So type eclipse is sent to draw eclipse, point to draw point etc.  Then each object can have its own value types which PlayerScreenUserInterface knows how to set, and output classes know how to decode.  If an output classes does not know the type, it simply just does not render it.  That way the output classes can be update as possible when a new object type is create (so at the start maybe the GDI output may not support the new mesh or magnifying class that Direct2D does).

25 Reply by joan (edited by joan 2010-11-09 22:37:40)

  • joan
  • joan
  • Admin
  • Offline

Hi,
My feeling is that some of these improvements are fairly independant and should be considered separately for clarity, even if they share the same goal. I'm not against completely re-writing the player screen, a good deep refactoring has been needed for far too long, but if it's possible to tackle these issues incrementally, it's better.
Having a buffer at input for example is not correlated to how the images are displayed.

In this regards, it may be more convenient to split the issues in different threads.
Maybe it's also time to move the discussion to the developpement mailing list… (And sorry Alexander for the thread hijack roll)

So if I wanted to identify independant threads:
1 - Adding an input buffer.
2 - Moving the decoder to its own thread.
3 - Move rendering code from the player control to a new rendering control.
3b - Make rendering controls from different screens use the same rendering surface.
3c - Make rendering control responsible for knowing how to draw specifc meta data objects.
3d - Adding Direct2D rendering control.

3b, 3c and 3d would be dependant on 3, but I think the first 3 could be coded independantly from one another.
I hope I didn't miss anything. What you refer to "drawing data class" is the "Metadata" class I think. It keeps all the things that might be drawn on top of the image.

I have a some more comments, but I think the discussions will start to be hard to track without splitting.
The mailing list page is here if that's fine with you…