ChatGPT writes as buggy code as a human Link to heading

The other day, I asked ChatGPT (v3.5) to generate a simple example code in C++ on interprocess communication via socket. The following is the server code generated from ChatGPT:

#include <iostream>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>

int main() {
    // Create a socket
    int serverSocket = socket(AF_INET, SOCK_STREAM, 0);
    if (serverSocket < 0) {
        std::cerr << "Error creating socket" << std::endl;
        return 1;
    }

    // Bind the socket to a specific address and port
    struct sockaddr_in serverAddress;
    serverAddress.sin_family = AF_INET;
    serverAddress.sin_addr.s_addr = INADDR_ANY;
    serverAddress.sin_port = htons(12345);
    if (bind(serverSocket, (struct sockaddr*)&serverAddress, sizeof(serverAddress)) < 0) {
        std::cerr << "Error binding socket" << std::endl;
        return 1;
    }

    // Listen for incoming connections
    if (listen(serverSocket, 1) < 0) {
        std::cerr << "Error listening on socket" << std::endl;
        return 1;
    }

    // Accept a client connection
    struct sockaddr_in clientAddress;
    socklen_t clientAddressLength = sizeof(clientAddress);
    int clientSocket = accept(serverSocket, (struct sockaddr*)&clientAddress, &clientAddressLength);
    if (clientSocket < 0) {
        std::cerr << "Error accepting client connection" << std::endl;
        return 1;
    }

    // Send a message to the client
    const char* message = "Hello from server";
    send(clientSocket, message, strlen(message), 0);

    // Close the sockets
    close(clientSocket);
    close(serverSocket);

    return 0;
}

And here is the client code:

#include <iostream>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>

int main() {
    // Create a socket
    int clientSocket = socket(AF_INET, SOCK_STREAM, 0);
    if (clientSocket < 0) {
        std::cerr << "Error creating socket" << std::endl;
        return 1;
    }

    // Connect to the server
    struct sockaddr_in serverAddress;
    serverAddress.sin_family = AF_INET;
    serverAddress.sin_addr.s_addr = INADDR_ANY;
    serverAddress.sin_port = htons(12345);
    if (connect(clientSocket, (struct sockaddr*)&serverAddress, sizeof(serverAddress)) < 0) {
        std::cerr << "Error connecting to server" << std::endl;
        return 1;
    }

    // Receive the message from the server
    char buffer[256];
    ssize_t bytesRead = recv(clientSocket, buffer, sizeof(buffer), 0);
    if (bytesRead < 0) {
        std::cerr << "Error receiving message from server" << std::endl;
        return 1;
    }

    // Print the received message
    std::cout << "Received message: " << buffer << std::endl;

    // Close the socket
    close(clientSocket);

    return 0;
}

I found two main issues with the code, a minor one that causes compilation error and a serious one that poses a security risk. Can you spot these issues?

The minor issue is the use of strlen() function without #include <cstring> macro in the server code. This is easy to catch, as the compiler will outright complain. The second issue is more serious, as neither gcc nor clang won’t warn you.

strlen() returns the length of a null-terminated string, excluding the null-character itself. In the case of the example, the message "Hello from server" contains 17 characters, so strlen(message) returns 17. Hence, the server is sending exactly 17 bytes of the string without the null-character itself. The client, however, assumes the data to be a null-terminated string. Thus, running the code may cause undefined behavior if the client’s buffer is not properly initialized to 0.

$ g++ -O3 server.cc -o server && g++ -O3 client.cc -o client
$ ./server& 
[1] 21750
$ ./client
Received message: Hello from servery�(<
[1]+  Done                    ./server

We can clearly see some garbage output from the client at the end of the message, since the message is not properly null-terminated. This can cause a serious security risk, since the program is reading memory address it shouldn’t. This case study proves that ChatGPT produces code that is as buggy as humans, probably because its train data contain such bugs.

The solution to ChatGPT’s buggy code? Well, we just have to be aware that the code produced by ChatGPT is not perfect. We should inspect the code as thoroughly as we would inspect human-written code.

The solution to this particular server/client example code? One simple solution from the server side is to send the string including the null character:

--- server.cc before
+++ server.cc after
@@ -39,11 +39,11 @@
 
   // Send a message to the client
   const char* message = "Hello from server";
-  send(clientSocket, message, strlen(message), 0);
+  send(clientSocket, message, strlen(message) + 1, 0);
 
   // Close the sockets
   close(clientSocket);
   close(serverSocket);
 
   return 0;

This way, the client can safely print out the message, since it is null-terminated.