Linux's Overcommit Allocator and OOM Killer

This is a topic that I have dealt with personally before, but I never found a good description of this. So I have decided to write about Linux's OOM Killer.

By default, memory allocation by applications in modern Linux kernels uses a scheme called overcommitting. What this means is that when an application requests memory, it virtually always succeeds. However, no physical page frames are actually allocated until that memory is 'touched' and a page has to be faulted in. Because of this, it is possible, and often likely, that the kernel has promised more memory to processes than is actually available and in the less likely case that they all have to use this memory at the same time, some process has to be sacrificed to free up memory.

There is a heuristical algorithm that decides which process to sacrifice. This algorithm is called the out of memory killer or oom killer for short. There are some tunables parameters and ways to avoid this seemingly-non-deterministic behavior.

The end result is that snippets such as the following might not terminate until well after the free number of pages have been allocated.

for(;;) {

  void *ptr = malloc(PAGE_SIZE);

}

If the memory were actually used, rather than leaked as with this trivial example, some process would have to be sacrificed in order to free some physical page frames (or ultimately, swap space). It isn't always the process with the most memory or a process trying to allocate memory at that time. That process could be a benevolent process that is behaving well but is unfortunate enough to be selected by the OOM Killer.

Disabling OOM-Killer per-process
Any particular process leader may be immunized against the oom killer if the value of its /proc//oomadj is set to the constant OOM_DISABLE (currently defined as -17).

Setting globally at runtime using the proc filesystem:
echo 100 > /proc/sys/vm/overcommit_ratio
echo 2 > /proc/sys/vm/overcommit_memory

Setting globally at runtime using sysctl:
This approach is a bit cleaner than just writing to the proc files.

sysctl -w vm.overcommit_ratio=100
sysctl -w vm.overcommit_memory=2

Setting globally at boot (persistent):
Add the following lines to /etc/sysctl.conf

vm.overcommit_ratio=100
vm.overcommit_memory=2

After you modify /etc/syctl.conf, the changes will be applied at the next reboot or you can run the following to apply the settings at runtime

sysctl -p

Notes

• All of these approaches set the commit ration to 100% (meaning don't overcommit) as well as turning the memory allocator to mode '2', meaning check if there is enough memory, and never overcommit.
• The amount of memory available to the allocator is the total physical RAM plus the total swap space, not just the RAM.

What a Difference a Decade Makes: Chemtrails

It seems like USA Today has come full circle on chemtrails. Here's a clip from an article in 2001 where they described the chemtrail phenomenon as a conspiracy theory.

Then just last month, Feb 2011, USA Today reintroduced the idea as a means for global engineering to combat so called man-made global warming.

This comes hot on the heels of the UN banning chemical trailing as a means of geo engineering. late last year

Murphy's Laws of Combat Operations

I have seen variations of these, but here are a few good ones. They are part tongue-in-cheek and part truthful.

Murpy's Laws of Combat Operations

• Incoming fire has right of way.
• If the enemy is in range then so are you.
• Don't look conspicuous - it draws fire.
• If it's stupid and it works, it ain't stupid.
• When in doubt - empty your magazine.
• The easy way is always mined.
• Try to look unimportant- they may be low on ammo.
• Teamwork is essential - it gives them somebody else to shoot at.
• Never draw fire - it irritates everyone around you.
• Anything you do can get you shot, including doing nothing.
• Never share a foxhole with someone braver than you.
• If your attack is going really well, it's an ambush.
• No battle plan survives contact with the enemy.
• Remember, all your equipment including your weapon was made by the lowest bidder.
• The only thing more accurate than incoming enemy fire is incoming friendly fire.
• When you have secured the area don't forget to tell the enemy.
• Make it tough enough for the enemy to get in and you won't be able to get out.
• If you're short on anything but the enemy - you're in a combat zone.
• The enemy invariably attacks on two occasions
• when you're ready and
• when you're not.
• A sucking chest wound is nature's way of telling you to slow down.

Liberal Smear Machine Backfires After Gunman Found To Be Occultist, Pot-Smoking Left Winger

http://www.infowars.com/liberal-smear-machine-backfires-after-gunman-found-to-be-occultist-pot-smoking-left-winger/

Verizon FiOS Remote Codes for Xbox 360

I found this to be useful. I didn't feel like shelling out a bunch of dough for the Microsoft brand remote when I already had the universal remote that came with my Verizon FiOS. They use a Phillips brand RC144 5302 remote (I've read the older one is 5301 and this code might work there as well). You can find the model number on a sticker in the battery compartment. Anyway, the remote code doc says to use code 0549 on the DVD setting. While my remote happily accepts that code on that device, it does not appear to control my Xbox 360 at all. Maybe that's for older Xbox 360 revision as my unit is only a year old as of this writing.

So I went to the user manual at http://onlinehelp.verizon.net/consumer/bin/pdf/Philips RC1445302 User Guide.pdf and noticed a peculiar device under Video-VCR that was branded Microsoft. I didn't know they made VCRs. I also saw Media Center PC under the Video - DVD section. They both had code 1999. In fact there was at least one other vendor with that code at a quick glance. So I programmed the remote:

1. Hold down the DVD button for 2 seconds
2. While still holding the DVD button, press OK. All the device lights should light up.
3. Enter 1-9-9-9 on the number pad. The DVD light should blink after a moment. If all the devices light up once for a longer time, then it did not accept the code. I don't know what to tell you other than try again in case you fat fingered it, verify the remote control model on the label, or try that 0549 code which didn't work for me.
4. Test it out!

Notes:

• This may work on the AUX device slot as well, I did not try it. However, I do know that some codes are not accepted on some device slots (that 0549 code entered correctly on DVD but not AUX, and it didn't work anyway but that's besides the point).
• The universal remote control's Power button turns on/off the xbox immediately. That is, there is no menu as when you hold the big Xbox Logo button on a wireless controller.
• The arrow buttons map to the D-Pad (or perhaps the stick) and function intuitively.
• The OK button seems to map to the Xbox A button, or at least the Xbox dashboard handles it in an intuitive fashion. I am not sure what the correct button is that maps to Xbox B button but Menu seems to kick you out of a movie you are viewing and puts you back at the Netflix menu.
• The Info button toggles the time display overlay at the top of the screen on Netflix.
• I have not tested the color buttons (the ones that Blu-ray devices seem to use) to see if they map to other Xbox buttons, but that would be icing since they are not really needed to watch movies.
• If you find any other useful buttons feel free to post them in the comments. Also if you have another model remote control from Verizon and found another code or that this code works too, please share.

Enjoy!

Why .NET Heralds the End of Intelligent Software Developers

These are sentiments I have felt for quite sometime, but have never written down, at least not in such length or breadth.

I believe that exclusive use of .NET is really causing the dumbing down of programmers. It's astounding how few people recently finishing their undergraduate degree know nothing at all about what is going on inside a computer or how their poor code or sloppy algorithms are thrashing the TLB, or D-cache, or harddisk or even have the slightest clue how to implement some code ouside of .NET. Don't even bother them with any kind of pointer math, algorithms, or data structures. I also feel that Microsoft's flagship language for the CLR, C#, is just a copy of Java that came 5 or 6 years later. Just compare the language rules, package framework, garbage collection, dynamic recompilation and bytecode (not exactly unique to Java either), and syntax of C# 1.0 and Java. It was basically Java with C++'s ":" operator for inheritance instead of Java's extends/implements keywords. Also, Generics were added in C# 2.0 which are just a rip-off, both syntactically and conceptually, of C++ templates except not as powerful (template methods and partial specialization not supported in 2.0 as far as I know for starters) and again 15+ years late to the game.

What do you get for all of the complexity? The most common pro I hear for C# v. C++ is garbage collection... if that's what you're looking for use RAII and stack allocated wrappers to clean up for you (e.g. auto_ptr, boost::shared_ptr, etc...). Have you ever seen a correctly written C# or for that matter Java loop exhaust the memory? Do you hate inserting artificial scopes (is that even possible in C#) or slow function calls in a loop to work around these issues? Don't you wish you could manage the allocation yourself sometimes? Too bad. The second most common is that you don't end up with NULL pointer dereferences... well you still end up with a similar number of "reference not set to instance of an object" errors which are exactly congruent and harder to guard against. In fact it's probably even worse because you cant check for equality with NULL first before dereferencing, except for the rarely-used "nullable types" which can then have the NULL pointer dereference issue again.

To draw an analogy, the creation of C# is kind of like the automatic transmission: most real auto racing is still done with some variant of a clutched manual transmission (except for some drag racing where shifting has to be really fast). Also I find that driving a manual transmission makes me more attentive, have more control over the vehicle, have a safer and easier merge onto the freeway (since I can choose when my transmission downshifts), and have a better chance and more responsiveness in the case of a crisis. In fact, if I had access to some accident statistics, we might find that drivers of manual transmission cars have fewer at-fault accidents per-capita (just speculating here, but would not be surprised). Thus, it's my personal experience that many more people knew at least something about cars and how they work in the past, versus now. In fact, if that manual shifting behavior was undesirable, you wouldn't see so many Manumatic systems in AT cars. You can argue that the car is "more accessible" now, but I argue that it is less flexible and it's users more lethargic or ignorant. That might boost sales, but I wager there would be few people still riding a horse if there were only MTs or if ATs were a special order options for those with a physical handicap preventing the use of MTs. Sorry to take so long to draw a parallel, but I think this is directly analogous to .NET. It's "a way" to do things, but it should not be "the way", is seldom if ever "the right way", and often is not even "a good way".

If you think that my disdain for .NET is an indication that I am some bitter old engineer from the '60s or '70s, I am actually in my 20s and thankfully just missed the fine line between when they stopped teaching "real" programming in school. And by the way, I used to work at Microsoft (full disclosure) and I left of my own accord, but not because of .NET :)

I originally posted this here, but I thought that I liked the response enough to post it on my personal blog.

Crash Windows Vista and Later Remotely With Non-authenticated User

Last month, I was looking into the reports of this flaw and I had written up a proof of concept in C/C++ on Windows. It seems pretty straightforward and I was able to crash Windows 7 Beta and RC. Due to the lateness of the exploits discovery, I would not be surprised if the exploit works on the Windows 7 RTM bits, or the next Windows Server bits. I don't have much time to explain the defect in detail, but suffice it to say that it relies on poor or just plain unvalidated SMB negotiation data in the srv2.sys driver. As most Windows driver authors know, crashes in kernel mode equals BSOD.

For more information, see the following links:

• One of the earliest reports AFAIK: http://seclists.org/fulldisclosure/2009/Sep/39
  
• Python version of exploit: http://g-laurent.blogspot.com/2009/09/windows-vista7-smb20-negotiate-protocol.html

For convenience (for most people), today I took the time to wrap the source code proof of concept in a VS 2008 project to make building the binary much easier. For people using some other build system, the only "non-default" lib you have to use is Ws2_32.lib, the Windows Sockets lib. For folks using GCC on Linux, sorry I did not have time to make this cross-platform or portable. It shouldn't be too hard for you to strip out the WSA* windows-specific sockets stuff and just use standard BSD-style sockets calls. There is also some address-resolution code that is probably MS-specific. In any event, the code is below. It's really just a single source file (though Visual Studio is good at adding a lot of crap to the project, I have tried to minimize it by not using PCH, etc...)

Here is a link to the project (includes binaries).

Notes

• DO NOT use this tool against someone's computer! This may be a violation of law in your jurisdiction. Please use this for academic purposes only, for rebooting your own machines, for generating crash dumps for novice investigation, or for generating crash dumps when debugging other drivers or system apps when you don't have an alternative method.
  
• The usage is pretty simple. From a command prompt just run "crash_remotely 69.69.69.69" where 69.69.69.69 is the remote machine address.
• The project builds, as configured, to use the MSVCRT. If you are not using VC 2008, you will need the MSVCRT 9 runtime.
• The target machine must be Windows Vista or later running the srv2.sys SMB/CIFS networking share driver, and have at least one active share (basically the driver must be loaded and processing connection requests). You do not need share access, which indicates the severity and exploitability of this bug.
• It is not clear when/if this bug will be addressed by Microsoft. If the tool does not work for you, tough luck.