A number of Windows and Linux GeForce 7 users have discovered that the command-line option –no_fbos improves their pixel-shader framerate a lot. Windows and Linux Radeon HD users have also discovred that –no_fbos cleans up artifacts in the water. Here’s what’s going on, at least as far as I can tell. (Drivers are black boxes to us app developers, so all we can do is theorize based on available data and often be proved wrong.)
Warning: this is going to get a bit technical.
FBO stands for framebuffer object, and simply put, it’s an OpenGL extension that lets X-Plane build dynamic textures (textures that change per frame) by drawing directly into the texture using the GPU. Without FBOs we have to draw to the main screen and copy the results into the dynamic texture. (You don’t see the drawing because we never tell the card “show the user”.)
We like FBOs for a few reasons:
- Most importantly, FBOs allow us to draw big images in one pass even if the screen is small. For example, if we have a 1024×1024 dynamic texture but the screen is 1024×768, then withou FBOs we have to draw the image in two parts and stitch it together. That sucks. With FBOs we can just draw straight to the texture and not worry about our “workspace” being smaller than our texture. This is going to become a lot more important for future rendering features where we need really-frickin’ big textures.
- It’s faster to draw to the texture than to copy to it.
- If you’re running the sim with FSAA, then we end up using FSAA to prepare all of those dynamic textures. In virtually all cases, we don’t need the quality improvements of FSAA, so there’s no point in taking the performance penalty. When we render right into the texture, FSAA is bypassed and we prep our dynamic textures a lot faster.
Since copying to a texture from the screen predates these new-fangled FBOs by several years, most drivers can copy from the screen to the texture very quickly; however we have hit at least one case where FBOs are much faster than copy-from-screen. That’s really a rare bug, and as you’ll see below, we see more weird behavior with FBOs.
When do we use FBOs vs. copying? Well, it’s pretty random:
- Pixel shader reflective water and fog use FBOs.
- Cloud shadows and the sun reflection when pixel shaders are off do not use FBOs.
- The airplane panel uses FBOs if the panel is 1024×1024 or smaller; if the panel is larger than 1024×1024 we draw from the screen and patch things together. So the P180 and the C172 are using different driver techniques!!
When you run X-Plane with –no_fbos, you instruct X-Plane to ignore the FBO capability of the driver, and we use copy-from-screen everywhere.
Mipmapping
There is one more element: mipmapping. A mip map is a series of smaller versions of a texture. Mipmapping allows the video card to rapidly find a texture that is about the size it needs. Here’s an example: imagine you have a building with a 128×128 texture. If you park your plane by the building, the building might take up about 100×100 pixels on the screen; your 128×128 texture is a good fit.
Now taxi away from the building and watch it get smaller out your rear window. After a while the building is only taking up 8×8 pixels. What good is that 128×128 texture? Its’ much too big for the job. With mipmapping, the card has a bunch of reduced-size versions of your texture laying around…64×64, 32×32,16×16, 8×8, 4×4, 2×2, 1×1. The video card realizes the building is tiny and grabs the 8×8 version.
Why not just use the 128×128 texture? Well, we’d only have two options with this texture:
- Examine all 16384 pixels of the texture to compute the 64 pixels on screen. That sucks…we’re accessing VRAM sixty four times for each pixel. Accessing VRAM is slow, so this would kill performance.
- Simply pick 64 pixels out of the 16384 based on whatever is nearby. This is what the card will do if mipmapping is not used (because option 1 is too slow) and it looks bad. Thsoe 64 pixels may not be a good representation of the 16384 that make up your building side.
So mipmapping lets the video card grab a small number of pixels that still capture everything we need to know about the building at low res.
We don’t mipmap our dynamic textures very often; the only ones that we do mipmap are the non-pixel-shader sun reflections and the pixel-shader sun reflections.
ATI
As far as we can tell, the current ATI Catalyst 8.1 drivers do not generate mipmaps correctly for an FBO-rendered texture. This is why without –no_fbos ATI users on Windows or Linux see very strange water artifacts. –no_fbos switches to the copy path, which works correctly.
At risk of further killing my track record of driver bugs in v9, we do think this is a bug. We have good contact with the ATI Linux driver guys so I have hopes of getting this fixed.
nVidia
It appears that the process of creating mipmaps for FBO textures is not accelerated by the hardware on the GeForce 7 GPU series. This is why GeForce 7 users are seeing such poor pixel shader performance, while GeForce 8 users aren’t having problems.
Now poor performance is not a bug; there’s nothing in the OpenGL spec that says “your graphics card has to do this function wickedly fast”. Nonetheless, what we’re seeing now is unusably slow. So more investigation is needed — given that the no-FBO case runs so quickly, I suspect the hardware itself can do what we want and it’s just a question of the driver enabling the functionality. But I don’t know for sure.
I get asked a lot about the limits of meshes and orthophotos in X-Plane. I’ll try to answer this, but the answer isn’t as simple as most people expect.
Texture Limits and Orthophotos
The maximum single texture size in X-Plane 8 is 1024×1024, and in X-Plane 9 it is 2048×2048.
I believe the maximum number of unique custom orthophotos that can be attached to a single DSF is at least 32768.
In practice, that number is pretty useless because X-Plane loads all textures for a DSF at the highest user-allowed res when the DSF is loaded. That means you tend to load a lot of textures. Every system is different and drivers have a lot to do with RAM efficiency, but generally you’ll run out of virtual address space and crash the sim before you can attach 32768x2048x2048 of pixels.
X-Plane has no limits on how the texturing is applied – that is, you can use your 2028×2048 texture to cover an entire tile or a single meter. So again, the limiting factor on the resolution of your orthophotos is how much total area you want to cover and how much RAM you can spend (remember RAM is also used for mesh complexity, 3-d models, etc.).
You do not need to have enough VRAM to hold all loaded orthophotos; the video driver will paeg the textures into VRAM. Virtual address space is the limiting factor. How far you push it depends on a lot of subjective things:
- If you expect your users to also run with a lot of trees, 3-d objects, cars on roads, and some plugins, you can’t use a lot of RAM.
- If you expect your users to have /3GB in their boot.ini and use nothing but your add-on, you can use a lot more RAM.
Generally the size of the DDS texture on disk is a good proxy for the virtual memory that is required to hold your textures.
It should be noted that these limits on texturing (due to X-Plane blindly loading a lot of stuff at once) affect all scenery types: objects, draped polygons, very complex airplanes, plugins, and not just terrain mesh orthophotos.
Getting Past the Texture Limit
It will take a future extension to the rendering engine to get past the current limits. Basically X-Plane will have to load textures at lower resolutions when they’re farther away. I don’t know when that is coming, but when it happens, it will increase the total amount of image data a DSF mesh can contain, because the limiting factor will be how much data is in the small area the user is looking at (since the rest can be stored at much lower res for far-away views). At that point the limiting bottleneck will be resolution (smaller means more data at once), not total image data.
Mesh Limits
Unfortunately, limits to the mesh are even more vague than limits to texture usage. X-Plane uses an adaptive mesh – basically you can put your vertices wherever you want. So the highest resolution you can achieve might be much smaller than 1 meter resolution, but you can only do this for a small area before the total mesh size gets too big. But this is okay – the intention of DSF is to let you put a lot of detail where you need it.
I believe that once again memory provides the first limitation to the mesh. That is – you’ll run out of memory loading your insanely huge mesh long before you hit a limit to the DSF container structure. And once again, even the RAM limit isn’t a hard limit because that virtual address space is shared with texures. Your mesh density limits actually go down when your textures go up because it’s a zero-sum game.
Estimating Memory
Here are some ideas on how to estimate your memory footprint:
- Run X-Plane over ocean to get an idea of the baseline memory use that the sim needs without extra scenery.
- Load your mesh without textures (move the textures away) to find the cost of the mesh itself. (I am going on the assumption here that you can rescale your mesh using whatever mesh generation tool you’re using).
- The size of DDS textures is a good proxy for the memory used.
Last night I created the “seed” files for MeshTool. Let me explain what these files are and why we’ll need them.
MeshTool is a wrapper around our irregular-mesh generation code. It will allow third parties to create base mesh scenery without having to create triangulations. Just like DSFTool saves people the work of having to encode DSFs (with point pools, command lists, and all that ugly stuff), MeshTool saves people the work of having to create their own triangulations.
MeshTool is a low level tool – you provide a text input file and some data. It’s designed to be an engine underneath tools like PhotoSceneryX, not an end in itself.
MeshTool will create “default” land-use terrain that approximately matches the global scenery, water, and custom orthophoto-based terrain. You (or a program you use) provide a text file that describes the boundaries between custom photos, land, water, and airports. You must also provide a SRTM-style HGT file for elevation.
How does X-Plane know what land-use should go on what terrain? That’s where the seed files come in. Our global scenery is generated from a set of rules that take into account morphology (land height and slope), approximate climate, and general land use. You provide the terrain shape via the HGT file, and we provide you with a seed file that contains climate and land use for that DSF tile.
Why do we provide the seed file rather than letting you find and create climate data? Well, our rules are tuned for a very specific pair of data sets; by providing the exact climate and land use data that we use, we assure that the rules files work correctly. The purpose of MeshTool is not to customize land use terrain, and we do not provide a mechanism for it. The purpsoe of MeshTool is to let you put orthophotos and new coastlines into the base mesh.
The good news is: seed files are tiny. They are typically 4 kb-8kb each; the entire data set is 322 MB total. That’s because the climate data is only 3×3 per DSF and the land use is only 121×121.
I hope to get MeshTool into some kind of testing within the next few weeks; if you are a programmer and would like to feed MeshTool from your own program, please contact me and perhaps I can arrange an alpha copy. I will also post the seeds as soon as I can.
Every now and then I see a comment in an X-Plane forum somewhere to the extent of:
“Joe Author made this great scenery pack for FS2K4, I tried to contact him about a port. I got no response, and the pack is free anyway, so I’ve posted my conversion.”
Simply put, you can’t do that, at least not in the United States. Copyright law is very clear on this subject: if you don’t hear back from the author, the default is that you do not have permission to create a derived work.
(The fact that the original package was “free”, meaning cost zero dollars, is not at all relevant. The author retains his rights to his own work even if he doesn’t charge money.)
A simple thought experiment reveals why it has to be this way: if I was giving away my new program as a promotional period and went on vacation, and you decided to post a derived work because (1) it was free and (2) you hadn’t heard from me, I would have no way to stop an illegal use of my work that I did not ever want (nor ever indicated that I wanted). “Free” plus “no one is home” is simply not a high enough bar to protect authors.
I may be fighting a pointless and unwinnable linguistic battle, but I have to try. People very often refer to the default city buildings in X-Plane as “auto-gen” but by any reasonable definition of “auto-gen” they are not really auto-gen at all.
Now these are all made up computer terms, so we can’t really check the dictionary. But “autogen” scenery (short for automatically generated) usually refers to scenery that is created by a flight simulator itself, usually while you fly, and usually by placing 3-d detail in places that match the base terrain. This exists in FSX, and existed in X-Plane up to version 7.63.
X-Plane 8 doesn’t have autogen!!!!!!! X-Plane 8 has scenery that is generated by computer programs, but X-Plane is not the computer program that is doing it. When you see a ton of buildings piled up in New York City, that is not becaues X-Plane looked at the New York city base terrain and said “hrm – some buildings would be nice.”
What actually happens is we analyze New York City when we create the global scenery (before we ever burn the DVD masters) and the DSF generator places all of those buildings in New York City. X-Plane simply gets a huge list of buildings from the DSF and draws them.
I am going to try to coin the term “algogen” (algorithmically generated) to describe these buildings that (like autogen) come from a computer generating semi-random buildings from input data, but unlike autogen, algogen is a process that runs once before the scenery is made.
So how is algogen and autogen different?
- You can’t change the pattern of algogen building placement by editing files in the sim. The algorithm has already been run! You can replace the buildings using an overlay (that excludes the base) or by using a library of models to substitute models.
- We are trading data size for computation. The DSF is bigger because it lists the location of every building in New York, even if they were just algogen buildings, but the job of placing those buildings is less difficult because X-Plane does not have to check each building against each road. That has been done in advance.
- Changing the scenery via an overlay doesn’t change the algogen! Add an airport via an add-on and you have to exclude the buildings. (But if you send that airport to Robin, the next global render will include it and the algogen will skip the airport automatically.)
Note one of the interesting results of algo-gen: X-Plane can’t tell the difference between an alg-gen building and a hand-placed one! They’re all just objects in a DSF. The fact that algo-gen buildings disappear with lower settings is because the sim/require_object property in the DSF header tells the sim which objects are important, and our generator always signals the buildings based on obstacle data as important. But algogen as a process is not visible to X-Plane!
And that’s why I’m spending so many words on trying to distinguish between “algogen” and “autogen” – because the processes are fundamentally different, they’re very different for scenery authors to work with. As a result, authors coming from X-Plane 7 or FSX will be very surprised if they try to understand X-Plane in terms of autogen….they won’t be able to find the autogen config files, and the autogen buildings won’t react to other scenery changes, because they’re not actually autogen at all!
Algogen is a classic pattern of “precompute” vs. “compute-while-fly”. Generally precomputing gives authors more flexibility (in our case, we have an obj engine that can handle a lot of objects, so authors can make their own objects of the same density as algo-gen with the objects placed anywhere) at the expense of making it more complex to edit the existing scenery (edit the mesh and the algogen doesn’t change).
When we started the v8 scenery, two things pushed me toward precomputation:
- In the past, changes in X-Plane’s rendering engine had broken third party add-ons. So a precomputation strategy (by getting the scenery code out of the sim) means that the sim is doing less “interpretation” and thus the interpretation of scenery is less likely to change.
- We wanted to focus on performance, which means getting computation out of the sim whenever we could.
Now that last point isn’t quite as important as it used to be…when we were doing the design (during mid X-Plane 7), dual core for everyone wasn’t on the radar, so the penalty for complex computation while flying is lower (and thus we have more expensive in-flight computation, like forests and completely draped bezier curve-based polygonal pavement).
But I think precomputation is still useful. Even with dual core, the algorithm that places X-Plane’s algo-gen bulidings can take one to two minutes for a 1×1 DSF tile on a very fast computer. That’s still a load time that’s out of the question for us; even on the second core, the DSF wouldn’t be “ready” in time for you to fly it. So one use of precomputation is to run algorithms that are more expensive than you can have in real-time. (That algorithm to pack objects inside an irregularly shaped polygon made by roads and land features is not fast.)
More importantly, precomputing does give us a nice advantage in the use of storage data. We ship about 50-60 GB of final scenery, but the source data is well over 100 GB. When we run the algogen algorithm, we have access to the full set of source data: coastlines, elevation, and land use before any simplification is done and any data is thrown out. So we have the potential not only to do a more complex analysis, but to do the analysis on a larger data set.
The down-side of precomputing is that if integration of all data is saved until sim time, there is the potential for third parties to contribute separate data to the sim via add-ons and still have the integration of those data sets work well. This doesn’t always work out – see complaints in online magazine reviews about combining orthophotos and new road grids in FS2K4…they don’t integrate because neither of those types of resources can be integrated to match the other in real time. But autogen still does a much better job than algogen at this; algogen basically has to be recut when other data changes. (And that is our intention – if you change the road grid, exclude and replace the objects too!)
The story on X-Plane performance is never over, but the chapter that is 9.00 pretty much is. I think we’ll be RC in the next build (if all goes well). Certainly a lot of the things that are still performance “problems” will require changes larger than we can do in a late beta.
I say problems in quotes because a lot of what’s been reported lately is in the form of: a huge screen res + a lot of shaders + a lot of FSAA = slow fps. That’s not really a bug, that’s an engine limitation. Now I want to make the engine as fast as possible, and a lot of this pixel shader stuff is new to 9.0, so if our track record for tuning stays the way it was for v8, we’ll probably get some efficiency improvements later.
But unfortunately there’s an underlying limitation: the new water and fog both cause the rendering engine to consume significantly more hardware resources than it would otherwise. Turn them on and you get prettier pictures at a price.
Just to post a a few general things I’ve found:
- X-Plane 9 will tell you where your GPU really stands. GPUs that were very adequate for X-Plane 8 (like the GeForce 6600 GT) will turn out to have nothing left in reserve for v9, while GPUs that were bored in v8 (the GeForce 8800 GTX for example) will show what it really has.
- Generally the cost of going from no shaders to shaders with water reflections of “none” and no volumetric fog should be very low if your screen res and FSAA don’t add up to something crazy (like 16x FSAA at 2048×2048).
- If you do have serious performance hits, try –no_fbos in the command-line; some drivers seem to have trouble with them.
- The P180’s virtual cockpit is a lot more expensive than the other ones, because it has a huge panel that is used in 3-d. We’ll hopefully rebuild the cockpit at some point.
- Turning water reflections to “complete” is very expensive. Watch the water and use the lowest setting that looks good. You don’t need complete reflections if there are a lot of waves!
- Shaders, FSAA, and screen size are all pulling from the same set of resources – be careful about cranking up all three.
- Check your v-sync – a lot of users whose vsync clamped them at 60 fps in v8 will be clamped at 20 in v9.
- Do your testing with texture res set low, then crank texture res later; pixel shaders also require the allocation of VRAM that can’t be purged (for things like reflection images) so running out of VRAM can show up in some weird ways performance-wise.
- The new Intel iMacs have serious performance problems with shaders on. This is due to driver limitations; given the much better performance under BootCamp, I expect the Mac performance to get better when the drivers are updated. For now I’d keep shaders off.
For now, please hold off on sending me performance reports. I just don’t have time to address them. In the future I will try to solicit very specific performance data points that we need to check. Perhaps in the future we can also set up a database of fps-test results to have a more comprehensive idea of how the hardware does its job.
I expect future features to appear in v9 that further eat hardware; those features will have an off switch. You may have to pick and choose what graphics you enable; there is no free lunch here. I also expect new graphics cards to emerge that make the GeForce 8800 GTX seem quaint!
What a difference new drivers make. ATI’s latest OpenGL drivers (Catalyst 8.1) seem to work quite well with X-Plane. On two fronts:
- Linux. Turns out all you need to do to make X-Plane happy on Linux with ATI hardware is update the drivers. I’m running with the Cat 8.1 drivers on my MacBook Pro and things look good. Use Catalyst 7.11 drivers or newer! No more MALLOC_CHECK_=1 or –no_threaded_ogl. With the next beta, you won’t have to use –force_run anymre.
- Windows. We were getting reports of corrupt screens on startup, and with the Catalyst 8.1 drivers these reports became very frequent. Turns out our threaded OpenGL code was doing something naughty*. Beta 19 fixes this.
The only known issue I can think of is: if you see corrupt water reflections, run with –no_fbos.
* Well, the way you set up threaded OpenGL on Windows and Linux is not very well documented, so I say naughty in that we made the drivers unhappy. I have yet to find a document that states clearly whether what we were doing is correct or not. We had to guess.
I figure there are three things that making blogging suited for software:
- A small interested group of users can subscribe to a blog without giving out an email address.
- The information in the blog can be found using a general search tool like Google.
- It’s easy for a busy programmer to post.
I think this last point is not to be ignored – I post a lot here because it’s easy enough to whip up a blog post that I can write one while the sim is booting.
The down-side of this is that blogs are not self-organizing. The blog is chronological, and somewhere within a heap of 200+ posts are detailed information on scenery topics not documented on scenery.x-plane.com.
That’s not good. So I’ll be trying to make a concerted effort to write real permanent documentation for some of the new scenery system topics that I cover. Documentation on DDS is in the works.
Part of the problem is that my interface for updating the X-Plane scenery website isn’t that robust. One of the nice things about the plugin system being a Wiki is that it’s easy to organize and easy to edit. (And one of my frustrations with “support forums” is that they don’t self-organize…they mix questions and answers based on history and not a search key that a user might use, like “what’s wrong with my card.” We’ll be supplementing the Linux forum with a Wiki soon.)
Here’s a summary of the new airplane features in 9.0 (and some coming). Hopefully this will give you an idea of what new capabilities are available for modeling planes in X-Plane 9. This list will sound like a broken record – virtually all of these features are optional; you don’t have to recut your finished airplanes to use them in version 9.
2-d vs. 3-d Panel
You may have noticed the new “3-d panel” option in PlaneMaker 9. This allows you to build a separate panel for the purpose of providing the texture to ATTR_cockpit (or ATTR_cockpit_region). You can then:
- Provide alternate instrument artwork in a cockpit_3d folder. (This lets you have perspective artwork for the 2-d cockpit and orthogonal artwork for the 3-d cockpit.)
- Pack your instruments together tightly to save space. (There is a real cost to large panels, so using a 1024×1024 panel for the cockpit object is a lot better.)
The 3-d panel is strictly optional, fully replaces the 2-d panel only for cockpit objects, and is activated by providing a custom panel background in a cockpit_3d folder. (See the “Example Plane-Widescreen+objects” plane in beta 19.)
ATTR_cockpit_region
Cockpit regions are an alternative to using the entire 2-d panel to texture your objects. They provide a few advantages:
- Performance. By requiring a power of 2 and allowing you to use a sub-area of the panel, cockpit regions avoid a lot of wasted computing that ATTR_cockpit can cause.
- Next-gen lighting. Unlike ATTR_cockpit, real 3-d lighting is applied to the panel when you use this attribute. This means that you will get a gradual decrease in light on your geometry (correct based on the angle of the sun) that matches the rest of the object.
Please note that you can mix and match which way you get your cockpit texture and whether you use the 2-d or 3-d panel feature (above) independently. However, you can only use ATTR_cockit or ATTR_cockpit_region in your airplane, not boht. ATTR_cockpit is still supported.
Generic Instruments
Generic instruments let you build instruments that follow some basic shapes (needles, tapes, etc.) that can be tied to any dataref. This both lets you customize particular instruments very precisely or create an instrument driven by a plugin dataref. These instruments are optional in version 9 – the old “premade” instruments are still supported.
New Datarefs
X-Plane 9 provides new datarefs targeted at airplane authors. The datarefs are better organized and have clearer names. But the old datarefs still exist, so legacy planes do not have to be updated.
Generally the entire cockpit should use only sim/cockpit2/ datarefs, and the plane exterior should use only sim/flightmodel2/ datarefs.
One special feature of these two sections: if your plane is used as an AI plane, these datarefs will animate the plane with the AI plane’s control deflections, not the user’s control deflections. So using these datarefs fixes the “AI animation” problem.
Plugins in Aircraft Folder
Version 9 airplanes may have a plugins folder (inside the ACF package) with fat plugins inside them. If you develop a plugin for your airplane, consider packaging it this way — this will allow your users to install the airplane with a single unzip for all platforms and no extra “drag-this-file-here”.
Plugins in the airplane folder is optional – you don’t have to provide a plugin, and plugins that are installed in the main Resources/plugins folder will still work. Still, I encourage you to use this feature because it makes the install process a lot simpler. The X-Plane SDK website will have documentation on fat plugins.
Liveries Folder
X-Plane 9 features a new “liveries” folder. Liveries (replacement exterior paint for airplanes and their attached objects) can be placed in packages in the liveries folder to greatly simplify the process of repainting an aircraft. See the “Example Plane-Widescreen+Objects” for an example.
While the liveries feature is optional, I strongly encourage anyone doing repaints to adopt it. Liveries can be switched by the user in the sim without any file manipulation; there is thus no risk of accidentally deleting or breaking an aircraft.
Large 2-d Panels
In X-Plane 9, a panel can be up to 2048×2048 in size. You pick the dimensions. The panel will scroll horizontally if necessary.
Note that if you use the new 3-d panel feature, the 2-d and 3-d panel do not have to be the same size. I would recommend a large 2-d panel (to fill large monitors) and a smaller 1024×1024 3-d panel (for performance).
Hiding Parts
X-Plane 9 will allow you to hide aircraft parts. Many v8 planes use OBJs to model the plane geometry, and use a transparent ACF texture to hide the ACF. Setting the parts to “not drawn” saves the CPU time that X-Plane would spend drawing the airplane, and is thus more efficient.
Keyframes
X-Plane 9 supports key-framed animation; this is useful for the scenery system, but for airplanes it allows for much more complex and realistic animation. OBJs that don’t have key frames still work.
Manipulators
This is a feature coming in the future: the ability to control how the user clicks and interacts with the cockpit object in detail. In X-Plane 9.0 we only support clicking on cockpit-textured geometry; manipulators will make features like draggable handles a lot more workable.
Global Illumination
X-Plane 9 does not yet offer a lot of control of the in-cockpit lighting environment; we’ll be working on this in future versions. These features will be opt-in…that is, you’ll have to change your model to get the new features, and old planes will work the way they always used to. It is likely that you’ll have to use “modern” airplane-building techniques to use these new features (meaning OBJs, named or custom lights, lego brick instruments ,etc.).
I try not to divert this blog from X-Plane related stuff, but I just can’t resist. Go to the itunes music store and search for:
amichai margolis
Ami has been a friend of mine for years; Orah V’Simchah is the first CD he’s recorded of his own music (I played bass on it). A few time a year he pulls a band together and we play a few gigs. Anyway, enjoy the tracks!
