2016 Jun 06

Pulling in slack messages for use on our labs page

Before we started creating our new labs page, one of the features we were looking for was Slack integration.

Here at Industrial, we use Slack on a daily basis to communicate with each other. It's a powerful tool that lets you converse in a number of ways. Some of the features we use involve reacting to messages posted by others with emojis, sharing files, starting group conversations and contributing to topic-specific channels which others are members of.

Integration of certain Slack messages to the website would satisfy two needs. One is to have up-to-date content on the website and second to provide a way to push updates about things we're working on without having to write a full blog post. Writing up a blog post or case study can take a lot of time. It is a better format when needing that extra verbosity. On the other hand, using Slack messages to share items that don't require that time investment is great. You can capture an idea within a few sentences or a link.

The original idea was to use the reactions functionality of Slack to upvote certain messages to the labs page. We decided that when we react to a message with a thumbs up emoji at least 3 times, it would appear on the site.

Using the Slack API, we were able to tap into the channel history endpoint to retrieve what we needed. This would give us the latest messages, but it would not do any other filtering that we were looking for. Only being able to show messages with the minimal amount of reactions is something that we had to do separately.

Planning the implementation

The basic requirements for building a page that listed out these Slack messages were as follows:

  • Use the Slack outgoing webhooks to let us know there is new content in a specified channel
  • Write a script to respond to the webhook being fired. In that script would be logic to retrieve and store the messages.
  • Output the pre-stored messages from the above script into the page.

We discussed how best to handle the retrieval and storage of messages. One way, we would request the Slack messages on every load of the labs page. That would have resulted in a lot of requests. The second option is to do the same but cache the output so it only runs when the cache is refreshed. This was better, but still not ideal.

The approach we decided on was to leverage the webhooks offered by the Slack API (as mentioned above) which lets us know when there's new content, which is the only time we want to use the API.
 
With the webhook approach decided, we weighed options on how to retrieve and process the messages. The idea was to have the webhook let us know of new messages, run a script to go get the messages, process them in a way that is ready for display, and then store that value.

Whenever someone would request the labs page, there'd be the minimal amount of work required to display the content.

This is the basic idea of how it is built. Now, we'll walk through the same path the code takes from posting in the #labs channel to displaying on the site.

From start to finish

Step 1 - posting a message

Post a message in the #labs channel. We created a dedicated channel for these messages to live in. We initially started with #general, but realized it wasn't ideal considering the crossover from other non-labs related posts.

Once there are messages appearing in that channel, we start reacting to them with "thumbs up", which translates to a "+1" under the hood for Slack. It takes at least 3 on any given message for them to be considered for display on the site. 

Step 2 - configure the webhook

There is an "outgoing webhook" setup in the settings of our Industrial Slack team. Webhooks are basically triggers that are provided to let API integrations know when something is happening.

They come is 2 flavors.

Incoming webhooks are used to interact with the service itself. In Slack's case, you'd be posting a message from an app to the account you're using.

The other type of webhook is outgoing.

It is there to provide outword facing notifications to apps you've built or for other services. In our case, we've setup an outgoing version to let our website know when new content is being posted in the #labs channel.

Configuration of such a webhook is simple. There's a drop down that tells Slack which channel to monitor, and once a message is posted, where to send the web request.

Slack uses the terms "incoming" and "outgoing" to specify its different notification types.

Typically other services provide an API and it is assumed that data as well as notifications can flow both ways without separate discussion of incoming or outgoing. For example, Mailchimp has its own API but also calls out webhooks separately. However, in their case, webhooks are meant to only work as outgoing.

As a baseline, webhooks are notifications. They are provided as part of a documented API to trigger code when things happen. We use Slack's API to retrieve data, but use the webhooks to tell us when. Without them we would be relying on some other interval to go and fetch data. Something along the lines of using the page refresh to go and get data.

There are a lot of great resources around the web for more info about webhooks

One final point to mention about our outgoing webhook. This is the conditions under which it will give us a notification. 
 
The Outgoing Webhook will only be triggered when one or both of the following conditions are met:

  • The message is in a specified Channel
  • The message begins with one of the defined Trigger Word(s)

We're really only concerning ourselves with the first one. Once we know there are new messages, we can then pull them in using our script. 

Step 3 - retrieve, parse and store the messages

A script on our web server pulls down and processes messages when the webhook is sent by Slack. This script is part of a custom module in Drupal and is the same one that we configured in the Slack webhook settings.

When new messages are available, the webhook fires it's notification at this file. It is this files job to go get the messages and filter them for messages that have more than 3 "thumbs up" reactions.

Since Slack doesn't offer this level of filtering when requesting messages, we just pull the latest 50 messages to have a sizable amount of messages to work with. The response comes back as JSON and so we can easily iterate over it. Messages that don't have the right amount of reactions will be thrown out and the latest 6 with 3 or more reactions will be stored in the database.

Filtered messages are still in the JSON format. We parse and convert them to HTML to be stored. For the most part, messages are markdown formatted so we use a markdown parser for PHP called Parsedown to convert most of the baseline text into HTML. Slack does use some proprietary ways of encoding certain parts of messages. For example, user mentions using the "@" sign come through looking like this <@U12345|rferguson>. For that, a separate custom parser was written to get the user's info using the slack API "users.info" method and print the real name post-parsing. There are a couple of different message types to deal with, file attachments/shares (which can be connected to a message) and there are Slack generated messages when people join channels etc.

For attachments, these are created when you share a link and it pulls in content from that shared webpage. We've written some logic to deal with this message content, and output it below the user's message on the site.

Channel joins type messages are ignored entirely. For files, we pull text from the associated content if provided, otherwise, file shares are private and so we're unable to display them publicly on the website. Therefore only the text was taken for these types of messages.

Once the messages have been parsed and converted to HTML for display on the site, they are stored using Drupal 8's state API.

This is a key/value store and we've used it here to capture the raw HTML we're going to put directly into the page. Storing the HTML within the all_messages variable looks like this:

// save the markup to state api
\Drupal::state()->set('slack_messages', $all_messages);

Step 4 - display on the site using a custom block

For actual display on the page, we've created a custom block that will pull out the markup from the state API. Since all the processing has been done beforehand in the custom webhook file, the block code is dead simple. Here's the entire content of that file:

<?php
/**
 * @file
 * Contains \Drupal\slack_block\Plugin\Block\YourBlockName.
 */
namespace Drupal\slack_block\Plugin\Block;
use Drupal\Core\Block\BlockBase;

/**
 * Provides slack block.
 *
 * @Block(
 *   id = "slack_block",
 *   admin_label = @Translation("Slack Block"),
 *   category = @Translation("Custom")
 * )
 */
class slack_block extends BlockBase {
  /**
   * {@inheritdoc}
   */
  public function build() {
    $messages = $this->get_slack_messages();
    return array('#markup' => $messages,
                 '#cache' => array('max-age' => 0)
                );
  }

  private function get_slack_messages(){
    $return .= \Drupal::state()->get('slack_messages');
    return $return;
  }
}

It pulls out the value and puts it to display. Caching of this block is turned off so that posts are displayed immediately.

The nice thing about the Slack webhook is that it fires very fast. Once content is entered into the #labs channel, the script is executed on our site in matter of seconds. The caveat being the upvote requirement for content to be displayed. Since we require a message to have a certain amount of reactions, and since the webhook doesn't fire on reactions being added, that means we have to wait for something else to be posted in the #labs channel before messages with 3 or more reactions can make it to the site. In other words:

  • Someone posts a message
  • Some people react to it
  • That message will still not appear on the website yet as the webhooks don't listen on reactions.
  • The next time a new message is posted to the channel (the webhook now fires), we then go get fresh messages, and we'll now see that there are new candidates which we can filter for with enough reactions.

This translates to a bit of a delay. That's the main reason the reactions are in place though. Otherwise, everything that would be posted would instantly be available to the website. Perhaps in the future Slack will expand the criteria upon which it listens and fires the webhooks. 

Conclusion

The concept of webhooks is a powerful thing in a world using API's to get things done. Their ubiquity is a great comfort when we think about how much we rely on the services we use to talk to each other in order to pass messages and activity from site to site (or service to service).

Most services you use and are familiar with employ poweful API's in concert with webhooks to trigger events like push notifications to your phone or update a webpage in real-time when something happens behind the scene. An entire platform exists, called Zapier, that leverages this idea to help you create automations based on timed events between the tools you use.

However, with Slack's API and webhook capabilities, we're able to provide a quick and easy way for our team to post messages to our site automatically (once enough votes are received) which allows us to share links and messages easily without going through the cumbersome process of writing a blog post.