Founder of LearnDataSci
How to Stream Text Data from Twitch with Sockets in Python
Learn how to connect to the Twitch Internet Relay Chat (IRC) using Sockets in Python and stream chat data for text analysis.
LearnDataSci is reader-supported. When you purchase through links on our site, earned commissions help support our team of writers, researchers, and designers at no extra cost to you.
You should already know:
- Python fundamentals – learn on Coursera
Twitch chat is a rich and interesting source of text data for NLP projects, but it's not entirely obvious how to get text from their API.
Web scraping would be one option, but fortunately for us Twitch offers a way to stream chat through IRC, which we can easily connect to using Python sockets.
Article Resources
- Get the notebook and scripts for this article on on GitHub
Getting your credentials
To stream messages from Twitch IRC you need to get a to token for authentication. To do that you need to:
- Create a Twitch account
- Go to http://twitchapps.com/tmi/ to request an auth token for your Twitch account. You'll need to click "Connect with Twitch" and "Authorize" to produce a token
Your token should look something like oauth:43rip6j6fgio8n5xly1oum1lph8ikl1 (fake for this tutorial).
Including your token, there's five constants we'll define for the connection to a Twitch channel's chat feed:
server = 'irc.chat.twitch.tv'
port = 6667
nickname = 'learndatasci'
token = 'oauth:43rip6j6fgio8n5xly1oum1lph8ikl1'
channel = '#ninja'
channel corresponds to the streamer's name and can be the name of any channel you're interested in. I chose Ninja because he is usually streaming every day for several hours and he has a lot of people watching him and chatting at once. So we rack up tons of text data quickly.
We'll stream one channel at a time to start, but towards the end of the article we'll create a class and command line arguments to watch multiple channels at once. Streaming multiple channels would provide us with some neat real-time data when we have a text processor in place.
Connecting to Twitch with sockets
To establish a connection to Twitch IRC we'll be using Python's socket library. First we need to instantiate a socket:
import socket
sock = socket.socket()
Next we'll connect this socket to Twitch by calling connect() with the server and port we defined above:
sock.connect((server, port))Once connected, we need to send our token and nickname for authentication, and the channel to connect to over the socket.
With sockets, we need to send() these parameters as encoded strings:
sock.send(f"PASS {token}\n".encode('utf-8'))
sock.send(f"NICK {nickname}\n".encode('utf-8'))
sock.send(f"JOIN {channel}\n".encode('utf-8'))PASS carries our token, NICK carries our username, and JOIN carries the channel. These terms are actually common among many IRC connections, not just Twitch. So you should be able to use this for other IRC you wish to connect to, but with different values.
Note that we send encoded strings by calling .encode('utf-8'). This encodes the string into bytes which allows it to be sent over the socket.
Receiving channel messages
Now we have successfully connected and can receive responses from the channel we subscribed to. To get a single response we can call .recv() and then decode the message from bytes:
resp = sock.recv(2048).decode('utf-8')
resp':[email protected] PRIVMSG #ninja :Chat, let Ninja play solos if he wants. His friends can get in contact with him.\r\n'Note: running this the first time will show a welcome message from Twitch. Run it again to show the first message from the channel.
The 2048 is the buffer size in bytes, or the amount of data to receive. The convention is to use small powers of 2, so 1024, 2048, 4096, etc. Rerunning the above will receive the next message that was pushed to the socket.
If we need to close and/or reopen the socket just use:
sock.close()Writing messages to a file
Right now, our socket is being inundated with responses from Twitch but we have two problems:
- We need to continuously check for new messages
- We want to log the messages as they come in
To fix, we'll use a loop to check for new messages while the socket is open and use Python's logging library to log messages to a file.
First, let's set up a basic logger in Python that will write messages to a file:
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s — %(message)s',
datefmt='%Y-%m-%d_%H:%M:%S',
handlers=[logging.FileHandler('chat.log', encoding='utf-8')])We're setting the log level to DEBUG, which allows all levels of logging to be written to the file. The format is how we want each line to look, which will be the time we recorded the line and message from the channel separated by an em dash. The datefmt is how we want the time portion of the format to be recorded (example below).
Finally, we pass a FileHandler to handlers. We could give it multiple handlers to, for example we could add another handler that prints messages to the console. In this case, we're logging to chat.log, which will be created by the handler. Since we're passing a plain filename without a path, the handler will create this file in the current directory. Later on we'll make this filename dynamic to create separate logs for different channels.
Let's log the response we received earlier to test it out:
logging.info(resp)Opening chat.log we can see the first message:
2018-12-10_11:26:40 — :[email protected] PRIVMSG #ninja :Chat, let Ninja play solos if he wants. His friends can get in contact with him.
So we have the time the message was logged at the beginning, a double dash separator, and then the message. This format corresponds to the format argument we used in basicConfig.
Later, we'll be parsing these each message and use the time as a piece of data to explore.
Continuous message writing
Now on to continuously checking for new messages in a loop.
When we're connected to the socket, Twitch (and other IRC) will periodically send a keyword — "PING" — to check if you're still using the connection. We want to check for this keyword, and send an appropriate response — "PONG".
One other thing we'll do is parse emojis so they can be written to a file. To do this, we'll use the emoji library that will provide a mapping from emojis to their meaning in words. For example, if a 👍 shows up in a message it'll be converted to :thumbs_up:.
The following is a while loop that will continuously check for new messages from the socket, send a PONG if necessary, and log messages with parsed emojis:
from emoji import demojize
while True:
resp = sock.recv(2048).decode('utf-8')
if resp.startswith('PING'):
sock.send("PONG\n".encode('utf-8'))
elif len(resp) > 0:
logging.info(demojize(resp))This will keep running until you stop it. To see the messages in real-time open a new terminal, navigate to the log's location, and run tail -f chat.log.
Parsing logs
Our goal for this section is to parse the chat log into a pandas DataFrame to prepare for analysis
The columns we'd like to have for analysis are:
- date and time
- sender's username
- and the message
We'll need to parse the information from each line, so let's look at an example line again:
msg = '2018-12-10_11:26:40 — :[email protected] PRIVMSG #ninja :Chat, let Ninja play solos'We can see the date is easy to extract since we know the format and can use the datetime library. Let's split it off and parse it:
from datetime import datetime
time_logged = msg.split()[0].strip()
time_logged = datetime.strptime(time_logged, '%Y-%m-%d_%H:%M:%S')
time_logged