Pthread in Real time kernel

[ThangLe]

Hi all,

i am using Fedora 16. after update the kernel to real time kernel:

Linux localhost.fedora 3.0.17-1.rt33.1.fc16.ccrma.i686.rtPAE #1 SMP PREEMPT RT Sat Jan 21 10:45:37 PST 2012 i686 i686 i386 GNU/Linux

I write a program to test kernel. whereby, i create 2 threads: Thread1 and Thread2 with different priority, Thread1 has lowest priority and thread2 has highest priority. Scheduler runs on FIFO policy.
In my expected, thread2 will run forever and will be not preempted by thread1 or stopped by kernel. However, actual result is both Thread1 and Thread2 have same chances to run.

/*
 ============================================================================
 Name        : Thread.c
 Author      : Thang Le
 Version     :
 Copyright   : Your copyright notice
 Description : Hello World in C, Ansi-style
 ============================================================================
 */

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

void *print_message_function(void *ptr);
int count1 = 0;
int count2 = 0;
int times = 1000;

int main(void) {
	pthread_t thread1, thread2;
	char *message1 = "Thread 1";
	char *message2 = "Thread 2";
	int iret1, iret2;

	//set attribute
	pthread_attr_t attr;
	struct sched_param parm;
	pthread_attr_init(&attr);

	/* Create independent threads each of which will execute function */

	pthread_attr_getschedparam(&attr, &parm);
	parm.sched_priority = sched_get_priority_min(SCHED_FIFO);
	pthread_attr_setschedpolicy(&attr, SCHED_FIFO);
	pthread_attr_setschedparam(&attr, &parm);

	iret1 = pthread_create(&thread1, &attr, (void*) print_message_function,
			(void*) message1);
	pthread_setschedparam(thread1, SCHED_FIFO, &parm);

	pthread_attr_getschedparam(&attr, &parm);
	parm.sched_priority = sched_get_priority_max(SCHED_FIFO);;
	pthread_attr_setschedpolicy(&attr, SCHED_FIFO);
	pthread_attr_setschedparam(&attr, &parm);

	iret2 = pthread_create(&thread2, &attr, (void*) print_message_function,
			(void*) message2);
	pthread_setschedparam(thread2, SCHED_FIFO, &parm);
	//set priority each thread
	//pthread_setschedprio(thread1, 20);
	//pthread_setschedprio(thread2, 1);
	/* Wait till threads are complete before main continues. Unless we  */
	/* wait we run the risk of executing an exit which will terminate   */
	/* the process and all threads before the threads have completed.   */

	pthread_join(thread1, NULL);
	pthread_join(thread2, NULL);

	printf("count1 = %d, count2 = %d\n", count1, count2);
	printf("Thread 1 returns: %d\n", iret1);
	printf("Thread 2 returns: %d\n", iret2);
	exit(0);
	//return EXIT_SUCCESS;
}

void *print_message_function(void *ptr) {
	char *message;
	message = (char *) ptr;
	while (times > 0) {
		printf("%s \n", message);
		if(strcmp(message,"Thread 1")== 0)
		count1 +=1;
		else count2 +=1;

		times --;
	}

	return (void*) NULL;
}

my expected:

count1 = 0, count2 = 1000
Thread 1 returns: 0
Thread 2 returns: 0

and actual result:

count1 = 540, count2 = 461
Thread 1 returns: 0
Thread 2 returns: 0

[Alberth5674764]

Assuming you have 2 (hyper) cores, why wouldn't both run?

[diamond_ramsey]

//set priority each thread
//pthread_setschedprio(thread1, 20);
//pthread_setschedprio(thread2, 1);

Is it possible to show both the values of the pthread_setschedparam(thread1, SCHED_FIFO, &parm); and pthread_setschedparam(thread2, SCHED_FIFO, &parm); after being assigned or at the end of this program?

If you run your program more than once, then same results?

If you changed thread1 to 1 and thread2 to 20, what results do you see and what results would you expect?

[ThangLe]

Originally Posted by Alberth5674764

Assuming you have 2 (hyper) cores, why wouldn't both run?

I run Fedora 16 in VMware 8, setting is: CPU x86 core 2 dual (processor: 1, number core: 2).

---------- Post added at 05:35 AM ---------- Previous post was at 05:26 AM ----------

Originally Posted by diamond_ramsey

//set priority each thread
//pthread_setschedprio(thread1, 20);
//pthread_setschedprio(thread2, 1);

Is it possible to show both the values of the pthread_setschedparam(thread1, SCHED_FIFO, &parm); and pthread_setschedparam(thread2, SCHED_FIFO, &parm); after being assigned or at the end of this program?

If you run your program more than once, then same results?

If you changed thread1 to 1 and thread2 to 20, what results do you see and what results would you expect?

I get many different results in many times run program, but very little of times gives the expected result.
if change thread1 to 1 and thread2 to 20, then result is till not as my expected (expected is only thread2 running, thread1 cannot run due to its low priority)
i attach source file for your reference.

[stevea]

Your understanding of priorities and scheduling is wrong.

The highest priority threads IN THE RUN STATE are executed.
With this kernel, if a higher priority thread enters the RUN SATE then it preempts the processor resource from any lower priority thread.
So IF thread2 can execute then thread1 cannot (assuming a single processor).

But whenever you make a system call there is some chance your process will BLOCK awaiting some resource. Then the thread enters a SLEEP STATE. For example if you make a 'read' syscall, then the read will block and not return until the read-data is available. Until then the thread is in a SLEEP STATE and cannot execute. Later when the read-data becomes avalable then the thread is restored to the RUN STATE, and then it will pre-empt anuy lower priority thread in the runqueue.

So while thread2 is in a SLEEP STATE then the low priority thread1 will run (assuming one CPU resource).

In your code 'printf' is repeatedly called, which eventually makes the syscall 'write' to the terminal. Doing a write can cause your thread2 to block and allow thread1 to execute.

-------------

Let me suggest this. Create a tmpfs or ramdisk file system (or use an existing one like /dev) and sent the output directly to this device ....

Thread > /dev/Thread.output

There is still some potential blocking in a ramdisk filesystem, but much less than on a disk or on a terminal.
So if you send the output to a ramdisk then there should be more thread2 and less thread1 executions.

---------- Post added at 03:33 AM ---------- Previous post was at 02:37 AM ----------

Sending output to /dev/null (revise to print stats in stderr) works ....

[stevea@crucibulum tmp]$ ./Thread >/dev/null
count1 = 0, count2 = 1000
Thread 1 returns: 0
Thread 2 returns: 0
[stevea@crucibulum tmp]$ ./Thread >/dev/null
count1 = 0, count2 = 1000
Thread 1 returns: 0
Thread 2 returns: 0


I also had to set the cpu affinity to force exectution on CPU zero.
Here are hte diffs.

-#include <stdio.h>
+#define _GNU_SOURCE
+#include <sched.h>
 #include <stdlib.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <assert.h>
 #include <pthread.h>
 
 void *print_message_function(void *ptr);
 int count1 = 0;
 int count2 = 0;
 int times = 1000;
+cpu_set_t *mask;
+size_t size;
+int numcpu=16;
 
 int main(void) {
 	pthread_t thread1, thread2;
@@ -23,6 +30,17 @@
 	char *message2 = "Thread 2";
 	int iret1, iret2;
 
+  mask = CPU_ALLOC(numcpu);
+  size = CPU_ALLOC_SIZE(numcpu);
+
+  CPU_ZERO_S(size, mask);
+  CPU_SET_S(0, size, mask);
+  if ( sched_setaffinity(0, size, mask) == -1 ) {
+    fprintf(stderr, "affinity failed\n");
+    exit(-2);
+  }
+
+
 	//set attribute
 	pthread_attr_t attr;
 	struct sched_param parm;
@@ -57,9 +75,9 @@
 	pthread_join(thread1, NULL);
 	pthread_join(thread2, NULL);
 
-	printf("count1 = %d, count2 = %d\n", count1, count2);
-	printf("Thread 1 returns: %d\n", iret1);
-	printf("Thread 2 returns: %d\n", iret2);
+  fprintf(stderr, "count1 = %d, count2 = %d\n", count1, count2);
+  fprintf(stderr, "Thread 1 returns: %d\n", iret1);
+  fprintf(stderr, "Thread 2 returns: %d\n", iret2);
 	exit(0);
 	//return EXIT_SUCCESS;
 }

[ThangLe]

Originally Posted by stevea

Your understanding of priorities and scheduling is wrong.

The highest priority threads IN THE RUN STATE are executed.
With this kernel, if a higher priority thread enters the RUN SATE then it preempts the processor resource from any lower priority thread.
So IF thread2 can execute then thread1 cannot (assuming a single processor).

But whenever you make a system call there is some chance your process will BLOCK awaiting some resource. Then the thread enters a SLEEP STATE. For example if you make a 'read' syscall, then the read will block and not return until the read-data is available. Until then the thread is in a SLEEP STATE and cannot execute. Later when the read-data becomes avalable then the thread is restored to the RUN STATE, and then it will pre-empt anuy lower priority thread in the runqueue.

So while thread2 is in a SLEEP STATE then the low priority thread1 will run (assuming one CPU resource).

In your code 'printf' is repeatedly called, which eventually makes the syscall 'write' to the terminal. Doing a write can cause your thread2 to block and allow thread1 to execute.

-------------

Let me suggest this. Create a tmpfs or ramdisk file system (or use an existing one like /dev) and sent the output directly to this device ....


There is still some potential blocking in a ramdisk filesystem, but much less than on a disk or on a terminal.
So if you send the output to a ramdisk then there should be more thread2 and less thread1 executions.

Thanks stevea,

I don't think i am wrong.
Let me explain: In the real time kernel, when a thread will go to Sleep status ?
Assuming that there is in single core.

In SCHED_RR scheduler :
There are 4 conditions to a thread go to sleep status:
1. It is blocked by accessing synchronize resource (I/O block, mutex, semaphore...)
2. or It is preempted by a higher priority thread.
3. or It volunteers to give up control of processor (call sleep() or pthread_yield()).
4. or Its Timeslice expired.

In SCHED_FIFO scheduler :
There are 3 conditions to a thread go to sleep status:
1. It is blocked by accessing synchronize resource (I/O block, mutex, semaphore...)
2. or It is preempted by a higher priority thread.
3. or It volunteers to give up control of processor (call sleep() or pthread_yield()).

in my code, even when i remove printf(), then result is not still as my expected.
And i found root cause is i am running program in muti Core, after i re-config my virtual machine to run with single core, then result is as my expectation.
addition, i need run program in root permission.

my code below shows these policies:

/*
 ============================================================================
 Name        : Thread.c
 Author      : Thang Le
 Version     :
 Copyright   : Your copyright notice
 Description : Hello World in C, Ansi-style
 ============================================================================
 */

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

void *print_message_function(void *ptr);

int count1 = 0;
int count2 = 0;
int times = 100000;

int main(void) {
	pthread_t thread1, thread2;
	char *message1 = "Thread 1";
	char *message2 = "Thread 2";
	int iret1, iret2;

	//set attribute
	pthread_attr_t attr1, attr2;
	struct sched_param parm1, parm2;
	pthread_attr_init(&attr1);
	pthread_attr_init(&attr2);

	//===============================================
	/* Create independent threads each of which will execute function */

	pthread_attr_getschedparam(&attr1, &parm1);
	parm1.sched_priority = sched_get_priority_min(SCHED_FIFO);
	pthread_attr_setschedpolicy(&attr1, SCHED_FIFO);
	pthread_attr_setschedparam(&attr1, &parm1);

	iret1 = pthread_create(&thread1, &attr1, (void*) print_message_function,
			(void*) message1);
	pthread_setschedparam(thread1, SCHED_FIFO, &parm1);

	//===============================================
	pthread_attr_getschedparam(&attr2, &parm2);
	parm2.sched_priority = sched_get_priority_min(SCHED_FIFO);
	pthread_attr_setschedpolicy(&attr2, SCHED_FIFO);
	pthread_attr_setschedparam(&attr2, &parm2);

	iret2 = pthread_create(&thread2, &attr2, (void*) print_message_function,
			(void*) message2);
	pthread_setschedparam(thread2, SCHED_FIFO, &parm2);

	//===============================================
	//set priority each thread
	//pthread_setschedprio(thread1, 20);
	//pthread_setschedprio(thread2, 1);
	/* Wait till threads are complete before main continues. Unless we  */
	/* wait we run the risk of executing an exit which will terminate   */
	/* the process and all threads before the threads have completed.   */

	pthread_join(thread1, NULL);
	pthread_join(thread2, NULL);

	printf("count1 = %d, count2 = %d\n", count1, count2);
	printf("Thread 1 returns: %d\n", iret1);
	printf("Thread 2 returns: %d\n", iret2);
	exit(0);
	//return EXIT_SUCCESS;
}

void *print_message_function(void *ptr) {
	char *message;
	message = (char *) ptr;
	while (times > 0) {
		//printf("%s \n", message);

		int i = 0;
		for (i = 0; i < 20000; i++) i++; // only for delay

		if (strcmp(message, "Thread 1") == 0) {
			count1 += 1;
		} else {
			count2 += 1;
		}

		times--;

	}

	return (void*) NULL;
}

result:

count1 = 100000, count2 = 0
Thread 1 returns: 0
Thread 2 returns: 0

as my expected

Anyway, thanks for your suggestion.