CSCI 201 System Programming

CSCI 201
System Programming

Spring Semester 2008

Welcome to CSCI 201; This page can be found on the Clark web server at http://babbage.clarku.edu/~jbreecher/System_Programming/

Basic Information
Course Description
Textbook
Lectures
Your Evaluation
The Projects

Basic Information

Professor:	Dr. Jerry Breecher, (jbreecher at clarku.edu)
	Office BP 343, (508)793-7396
Meeting Time:	Lecture: Monday, Thursday 2:50 p.m. - 4:05 p.m.

Office Hours:	Monday, Thursday 10:00 - 12:00
	By appointment; 24 hour response by email

CSCI 201 has the general designation "Pro Seminar" Under this title, a number of possible topics have been taught over the years. This year we're offering System Programming - how to program with an awareness of the computer and the world around you. This can be programming of requests to the Operating System, to the Network, to a video screen, etc. Many of these programs have considerable knowledge of the hardware and software in their environment.

Textbook

Recommended:

	LINUX
Harold, Elliotte	Java I/O, 2nd Edition	O'Reilly, 2006	ISBN-10: 0-596-52750-0
Johnson, Huizenga & Pulavarty	Performance Tuning For Linux Servers	IBM Press, 2005	ISBN-10: 0-13-144753-X
Johnson & Troan	Linux Application Development, 2nd Edition	Addison Wesley, 2005	ISBN-10: 0-321-21914-7
Love, Robert	Linux System Programming	O'Reilly, 2007	ISBN-10: 0-596-00958-5
Stevens & Rago	Advanced Programming in the UNIX Environment, 2nd Edition	Addison Wesley, 2005	ISBN-10: 0-201-43307-9
Rochkind, Marc	Advanced UNIX Programming, 2nd Edition	Addison Wesley, 2004	ISBN-10: 0-13-141154-3

	WINDOWS
Hart, Johnson	Windows System Programming, 3rd Edition	Addison Wesley, 2005	ISBN-10: 0321256190

Lectures

Each class session will be devoted to a combination of "Show and Tell" - looking over your work produced for the class, and a lecture/demonstration of a new programming technique.

The Projects

During the semester there are 28 lectures. For all but the first class, there will be a program assigned and submitted. These will be fairly short each time, but freqently assignments will build on the work of a previous assignment.

Your Evaluation

Your evaluation in this course is determined by:

100% Programs you Submit - your grade will be the highest 24 of 27
0% Exams

Course Events

Operating Systems	LINUX	Windows
Languages	Java	C / C++
IDE	Eclipse	Visual Studio

Directories, Files and IO

These assignments use Linux, Eclipse, Java,

Lecture	Subject	Assign	Harold	Love	Johnson	Rochkind	Description Due
1 - Jan 14	Environments / File IO - Streams		2, 3	2	11	2
2 - Jan 17	File IO - Streams	Prog02	4	2	11	2	Write a program that copies a 1 megabyte file as fast as possible.
X - Jan 21	No class
3 - Jan 24	File/Dir Mgmt	Prog03	17	7	14.3	3	Extend fileDumper on pg 53 to output also octal and binary
4 - Jan 28	File/Dir Mgmt	Prog04	17	7	14.7	3	Find number of subdirectories that can be created, one under another.
5 - Jan 31	File/Dir Mgmt	Prog05	17	7	14	3	List attributes (size, dir/file, etc.) of all files in current dir
6 - Feb 04	File/Dir Mgmt	Prog06	18	7	14	3	Create a tool that behaves equivalent to 'du' on Linux.
7 - Feb 7	Threading/Locking/Flushing	Prog07	15	3	13.3	2	File Locking.
8 - Feb 11	File IO - Buffered	Prog08	14	3	13	2	File Locking
9 - Feb 14	File IO - Buffered	Prog09	14	3	13.2	2	File Locking
10 - Feb 18	File IO - Nonblocking	Prog10	16	3	13.1	2

Processes: Characteristics, Management, Synchronization

These assignments are Windows, Visual Studio, C

Lecture	Subject	Assign	Hart	Love	Johnson	Rochkind	Description Due
11 -Feb 21	Environments	Prog11	Ch 6				Sort a large file
12 -Feb 25	Proc. Stats	Prog12	Ch 6				CreateProcess
13 - Feb 28	Proc. Stats	Prog13	Ch 6				Process Characteristics
X - Mar 3,6	Spring Break
14 - Mar 10	Threads & Scheduling	Prog14	Ch 7				More on CreateProcess
15 - Mar 13	Threads & Scheduling	Prog15	Ch 7				Creating Threads - how do threads interact with each other?
16 - Mar 17	Synchronization	Prog16	Ch 8				Test driving synchronization Mechanisms.
17 - Mar 20	Synchronization	Prog17	Ch 8, 9				Now that we have synch primatives, let's use them
18 - Mar 24	Pipes between processes	Prog18					Now make synchronization work between processes.
19 - Mar 27	Signals / IPC	Prog19
20 - Mar 31	Signals / IPC	Prog20

Memory, Time, Security These assignments are Linux, Eclipse, C

Lecture	Subject	Assign	Hart	Love	Johnson	Rochkind	Description Due
21 - Apr 3	Memory	Prog21		Ch 8	Ch 7
22 - Apr 7	Memory	Prog22		Ch 8	Ch 7
23 - Apr 10	Memory	Prog23		Ch 8	Ch 7
24 - Apr 14	Time	Prog24		Ch 10	Ch 18	Ch 9
25 - Apr 17	Time	Prog25		Ch 10	Ch 18	Ch 9
26 - Apr 21	Security	Prog26		Ch 2	Ch 22
27 - Apr 24	Security	Prog27		Ch 3	Ch 22
28 - Apr 28	Last Show & Tell
	No Final Exam

Program Specifications:

Prog02	File copy: You will be given a 100 megabyte file. Your task is to read the data from that file and write it into a new file. Figure out a way to time the duration of the copy. Usage: "java MyFileCopy Source_File Destination_File"
Prog03	It's often necessary to view the contents of a file in a number of possible formats. Extend the program FileDumper.java on page 53 of Harold so that it also prints out the data in octal and in binary format. In addition, the format should look like this printout from od 0000000 147720 160021 130641 160432 000000 000000 000000 000000 0000020 000000 000000 000000 000000 000076 000003 177776 000011 0000040 000006 000000 000000 000000 000000 000000 000001 000000 0000060 000163 000000 000000 000000 010000 000000 000165 000000 0000100 000001 000000 177776 177777 000000 000000 000164 000000 0000120 177777 177777 177777 177777 177777 177777 177777 177777 * 0001000 000017 001750 005720 000000 000001 001751 000050 000000 OD has these features: 1. The first column is the location of the data in the file - in the example this is the octal address but the base for the address should be specifiable as decimal, hex, or octal. 2. The spec currently defines "Usage: java FileDumper [-ahd] file1 file2..." where a, h, and d ask for the data to be written in ASCII, hex, or decimal. You will be extending this so the data can also be dumped in octal and binary. The overall specification will then be "Usage: java FileDumper [-OHD] [-ahdob] file1 file2..." 3. Use some common sense in the amount of data you have per line. No more than 80 characters/line; make octal be multiples of 8 bytes/line, decimal 10 bytes/line, hex 16 bytes/line. 4. When data is duplicated for the whole output line, do not print it but instead indicate that repeat with a "*" as shown in the od output.
Prog04	How many directories can you create, one under another? It's equivalent to the shell commands mkdir foo, cd foo, mkdir foo, cd foo and so on until something breaks. You must also devise a way to remove these directories. WARNING: It can be very difficult to remove directories 1000 levels deep - LINUX is not very good at that operation. So make sure your program can do the removal also. Design the program with a max-level control so your deletion goes more easily: "java MakeDirs Max-Directory_Depth"
Prog05	Write a program that lists all the properties of the files in a specified directory. Here's an example of the program I wrote in C to do this. Java may be able to list more or fewer properties. Usage: dir2 <Directory> dir2 . -rw-r--r-- 1 jbreecher users 949 May 20 2003 extio.h drwx------ 4 jbreecher users 4096 Jan 24 10:19 .. -rw-r--r-- 1 jbreecher users 136 Feb 24 2003 macrostr.h -rw-r--r-- 1 jbreecher users 219 Jun 22 2003 logf.h drwx------ 3 jbreecher users 4096 Jan 27 2008 . -rw-r--r-- 1 jbreecher users 1820 Jan 23 13:20 dir.c -rwx------ 1 jbreecher users 12595 Jan 23 13:44 dir2 Some additional thoughts: How would you go about making this a sorted list? The secret to gathering the file characteristics on LINUX is the stat command. Can you figure out how to call a C function from java; this gives you the advantage of both worlds - the nice printing facility of java and the native ability of accessible LINUX system calls.
Prog06	Extend the Prog05 to achieve two functions. Function 1: A program that can descend throughout a directory & subdirectories and find the total number of blocks in all files. This is similar to the functionality of "du". However what generally happens is a need for a tool that can find the biggest files -- If you're space conscious enough to run du, you'd like to know what can be eliminated. So some means of highlighting the biggest files and the biggest directories helps a user focus on the files that can be targeted. Function 2: A program that can find requested text in any file within a given directory and sub-directory. This is a grep program but it works on more than the current directory. Usage looks like this: megagrep <search string> <dir and sub dirs to be searched> megagrep 'the following' . suvreq.h: of the following symbols must be defined (1003.1-1988 isn't supported): /subdir/another_file the following people a) The string to be searched can be put in single quotes (linux will treat this as a single argument for you) b) You can ignore files that are binary or don't have printable characters in them.
Prog07	I have written a producer program than feeds records into a database. In class we will go over how this database is laid out. It has an In-Use record, a Header containing pointers to doubly linked lists, and the data file itself, being a sparse set of the produced records. In this first part of the project, your task is to build your own Producer.java and also a Consumer.java of these items. I wil provide you with my Consumer.c and my Display.c so that you can inspect the files that result.
Prog08	Continue with Prog07
Prog09	I expect that concurrently running a Producer as well as multiple Consumers will result in a wild mess. With no locking on the various portions of this database, actions aren't atomic, so the records will very quickly become inconsistent. Your task in this Program is to introduce locking of the files so as to maintain consistency.
Prog10	Continue with Prog09
Prog11	You are given a very large file (100 megabytes or larger) made up of records of a given size. Your job is to sort those records as quickly as possible. If you do it by reading two records sorting those two, and writing them back to the disk, it will take forever. The only way to do this efficiently is to read the file into memory and sort it as an array of records. This is explained on pg 323 of the Java IO book under the section on memory mapped IO.
Prog12	In directory http://cs.clarku.edu/~jbreecher/System_Programming/Prog12/ are a number of programs. Some run on LINUX, Some run on Windows. Here's a table showing what you should do: Source Code Use gcc on LINUX Use gcc on Windows Use Visual Studio on Windows hello.c Yes Yes Yes runit.c Yes Yes No CreateProcess.c No Yes Yes To load gcc on your Windows Box, Get the Automated MinGW installer at http://sourceforge.net/project/showfiles.php?group_id=2435 Then follow directions. You must set the path so your command window can see the gcc - this is done via start->control panel->system->Advanced->environment variables. I also like a gdb so a windows version is at http://sourceforge.net/project/showfiles.php?group_id=2435&package_id=20507 - select current version rather than one of the candidates. Your task for next class is simply to get this all running. Submit the .exe for the three programs. Show that you can use Visual Studio. The assumption is that you will have an environment up and running for continuing work on Windows.
Prog13	To give you practice using structures in C, please write a program that uses the following system calls and prints out all the possible information returned by these calls. GetProcessIOCounters GetProcessMemoryInfo GetProcessWorkingSetSize GetProcessTimes
Prog14	Your task is to write a program that creates multiple processes - in fact, we'll want to test it to see how many processes can be started on a Windows box (it's not that many since they get bogged down easily.) This will require a program that can pass command arguments to the child program. It may also require that the child program be able to read those arguments in a non-standard way (don't know about that at this writing.) Your task is to build a program that emulates this bat file: @echo off REM This is a script to run numerous processes. REM @set debug=0 if "%debug%" EQU "1" @echo on echo. REM REM Make sure we got the parameters we expected - othewise give instructions. REM if "%2" EQU "" ( echo This script expects 2 arguments: echo 1. The number of processes to create echo 2. The sleep time in each process loop goto EXIT_HERE ) REM REM Evaluate the arguments REM set /A NumProcs = %1 set /A SleepTime= %2 REM set /A Counter = 0 if exist trigger del trigger echo "This is the trigger file" > trigger REM Now start up the required number of processes :Loop echo The counter is %Counter% start/min /belownormal ManyProc.exe %SleepTime% "trigger" set /A Counter = %Counter%+1 if %Counter% LSS %NumProcs% goto Loop :Exit_Here When I ran the CreateProcessA system call, it was difficult to tell if the processes were actually created and running. You can determine this by running the task_manager and looking at the processes tab.
Prog15	You are given two starter programs, both of which work in their environments. WindowsStarterCode.c is a great example of how to create a thread in a Windows environment. LinuxStarterCode.c is the functionality we'd like to perform. Your task is to merge the two programs in a Windows environment so that your final program has an Adder thread and a Subtracter thread, both of which are operating on a global variable. The behavior of WindowsSolution.c should then be approximately the same as the behavior of LinuxStarterCode.c.
Prog16	As a result of completing Prog15, you produced a program that has no locking! Since the adder and subtracter are asynchronous and have no locking around global variables, there's no guarantee that the update of the global variable is atomic. For instance, here's the code for the adder. (I've made a few changes by putting in a few more global variables. DWORD WINAPI Adder(LPVOID lpParam) { int i, loopcnt; PMYDATA pData = (PMYDATA)lpParam; loopcnt = pData->val1; for (i = 0; i < loopcnt; i++) { GlobalVariable1++; // Protect this variable using a Semaphore GlobalVariable2++; // Protect this variable using a Mutex GlobalVariable3++; // Protect this variable using a Windows Critical Section object GlobalVariable4++; // Protect this variable using an Event } // End of for Sleep( 1 ); return(0); } // End of Adder In most cases, the general format of these devices is: 1. Get a lock (be guaranteed that you have exclusive hold on a area) 2. Modify a shared resource knowing you are the only one having access to it. 3. Release the lock. The result of these tests SHOULD be that at the end of the run, each of these variable has been updated atomically and they each equal 0. NOTE WELL: When we did locking before, there was lots of confusion. For instance, a Critical Section is an "area" of code - in this case, it includes the TWO places that touch GlobalVariable3. This means that you should use only ONE ONE ONE critical section variable, initialized in main, that is seen and used by both adder and subtracter.
Prog17	Your task is to synchronize producer(s), a consumer(s) and reader(s). These threads interact in this fashion: Producer(s) --> PRODUCER LINKED LIST --> Consumer(s) --> DONE LINKED LIST --> Readers(s). The rules for synchronization look like this: Producers: Must have exclusive access to the PRODUCER LIST in order to write to it. Consumers: Must have exclusive access to the PRODUCER LIST in order to modify it. Must have exclusive access to the PRODUCER LIST in order to write to it. Readers: Do NOT need exclusive access to the DONE LIST in order to read it. BUT if there are any writers, then they can not read. It does mean there can be other readers at the same time, just so there are no writers. I have given you the starter code with a mechanism for the producers - the producers need locks. And then the consumers and readers code needs to be written.
Proposal	The Project: Write an e-mail application that sends and receives e-mail and attachments. Language: Java Why: Java can send and receive e-mail directly to/from the internet. New Tools: SMTP MySQL Project 1: Write a Java e-mail client that sends e-mail to existing e-mail addresses, through the command line or through a browser/GUI for the ambitious. SMTP is the tool of choice here. Project 2: Add to the program so that it can listen for and receive messages for a particular e-mail address. This appears to be the most difficult part of the project. Note that as there is no way to store said messages, therefore once they are displayed they may disappear. That comes next. SMTP is again used. Project 3: Formulate a way to store user information (i.e. username/e-mail address, password(?) and messages for ONE user only (for now). Tool: MySQL (since it can communicate with Java). MySQL is a very simple database language that needn’t be fully learned to implement basic databases. Project 4: Flesh out the user interface a bit more: options, folder display, navigation Project 5: Attachments. Make them work (again for one user). Project 6: Make multiple users an option, i.e. with a log-on screen, password, and restrict a user to only their own files. Project 7: Polish the entire project (if needed)- make it shiny and useable and flawless.
Prog18	In real life, multithreaded programs can be tricky. In the typical software shop, development of a large application is most likely to use processes rather than threads. This is because developers working on various parts of an application won't collide with each other as they would if everything is in the same process. "Collision" in this case means both with regard to code stepping on someone else's code (isolation) and work scheduling - independence of product makes it much easier to ship a product. In Program 17, you developed a threaded application where the various threads communicated via shared memory (linked lists) and some synchronization mechanism. This is probably the best-performing mechanism, but is subject to deadlocks, etc. So for Program 18 you are to develop the same functionality as the last assignment, but with separate processes. This code will "feel" the same as the last assignment. > Prog18 Usage: Prog18 <Items Produced>" The producers, consumers, and readers communicate with each other via pipes. You will find this is MUCH easier to do than maintaining linked lists and synchronization (but is much slower). Hints: 0. I was originally going to specify multiple producers, consumers, readers, but I have had an awakening and value your sanity this weekend. 1. Useful commands are CreateNamedPipe and CreateFile. 2. A message type pipe will be a lot easier than a byte pipe. 3. Writers should use PIPE_NOWAIT so they can continue to fill the pipe even if noone is ready to read. They will need to be able to handle an error if the pipe is full (how do you set the size of a pipe???) 4. Readers should use a PIPE_WAIT so that if there's nothing to do, they can simply let the OS handle waiting around.
Email Server Assignment for March 27	Project 1: Write a one-sided server which sits on port 20120 and waits to accept incoming communication. It should print out the received message if it successfully makes a connection with another program. You will receive a Java class file for a client which ought to work with your program. All the client does is attempt to establish a connection with anything listening on port 20120 of specified machine and if successful, it sends a message and closes all connections. This program is NOT a long one, probably no longer than twenty or thirty lines- the hurdle lies in learning the new calls and terms and understanding how it ought to work. ServerSocket class (look at Accept method plus InputStreamReader ) Here's the picture as it stands now:
Prog20
Prog21
Prog22
Prog23