A Python package for tokenising Japanese-language tweets

Last time I wrote about preparing Japanese-language Twitter data for machine learning purposes, I said I’d likely come back to the topic and discuss different approaches to using this data to find out useful or interesting things.

I will. Eventually. I’m working on a number of interesting projects at Waseda at the moment, using several different machine learning approaches (some supervised, some unsupervised) to cut through the noise of Twitter data and find out (hopefully) relevant political or sociological things. I’ll write up some practical guides to those approaches as soon as I have time.

In the meanwhile, though, I’ve battle-tested a lot of the techniques outlined in the previous blog post, fixed some bugs and updated a handful of things to be faster or more effective. I think I’ve now got a pretty solid system in place for processing Japanese-language tweets and splitting them down into handy bags-of-words – without losing semantically important features like emoji or kaomoji along the way.

Rather than update the last blog post, I’ve bundled all of the various functions into a Python object that you can use in your own code, and uploaded it to Github. You can find the code here. It’s not the world’s most professional Python code and for now at least, you’ll need to just download the file and drop it into your project – I haven’t got around to turning it into a “real” Python package yet – but it will hopefully be useful to some people working on this kind of project nonetheless.

Corbyn’s manifesto is great. Labour is still going to lose.

Corbyn Labour Coup

The UK’s Labour Party yesterday announced its manifesto for the upcoming general election. It’s a dramatically different vision to the path the Conservative party has laid out for the country, and unsurprisingly, it is the most left-wing manifesto the party has had since the rise of “New Labour” under Tony Blair in the late 1990s. Contrary to the scoffing of some on the right, however, it’s far from being a loony-leftie document that appeals only to the fringes. If anything, it’s an incredibly populist manifesto; the majority of its policies enjoy very broad support from the British electorate.

The right-wing press argues that the manifesto shows Labour, under Jeremy Corbyn, trying to return the UK to the “bad old days” of the 1970s (yes, that’s the same right-wing press that’s had a months-long priapism over notions like returning to Imperial weights and measures, or issuing blue-coloured passports; the existence of irony, it seems, is just another EU strategy to undermine Brexit). To some extent, they’re right; the manifesto does hark back to the pre-Thatcher years in parts, with policies aimed at undoing some of the more egregious mistakes of the neo-liberal policy regimes of the past 35 years. These include the ideologically motivated privatisation of a number of natural monopolies like public transport and energy, or the underhanded social engineering that saw council housing being sold off cheaply and never replaced, both of which date back to the early 1980s and are targeted in the manifesto.

Rather than arguing back and forth about the benefits of various different aspects of the manifesto, though, the point I want to make is that regardless of whether you consider these policies to be economically sensible or politically desirable, they’re undeniably popular. Opinion poll after opinion poll has shown – with margins that defy post-Brexit, post-Trump scepticism about polling – that the British public support the renationalisation of rail and other core services, want to see council housing stocks replenished, and favour the roll-back of the most extreme deregulations of the labour market, such as zero-hours contracts. If you go through the Labour manifesto line-by-line with British voters, you’ll find a strong majority in favour of pretty much every major policy in the document. The last manifesto to enjoy such a strong level of support was probably Blair’s in 1997 – a very different manifesto for a very different time.

Blair won 1997 in a historic landslide. Corbyn, for all that his policies resonate, is going to lose, and lose badly – likely handing Theresa May’s Conservatives a significantly boosted majority in the House of Commons, and perhaps losing key seats once seen as Labour strongholds. This is in spite of the fact that May’s Conservative policies are actually pretty unpopular; “Hard Brexit” is opposed by a plurality of the electorate, and some of her policies around things like education, the NHS and fox-hunting (yes, the fox-hunting debate is back) are opposed by a significant majority. It doesn’t matter; she’s going to win the most convincing Conservative electoral victory in a generation.

What this means, from a political science wonk perspective, is that a significant part of the British electorate is going to go out and vote for a party whose policies they disagree with. It flies in the face of certain fields of theory, which try to link the policy preferences of voters to their choices in elections, or to model the behaviour of candidates as principal-agent relationships – in which voters (principals) elect candidates as their “agents”, who go on to represent the policy interests of the voters in order to ensure future re-election. There’s more complexity to those models, but in essence they all assume the same fundamental thing – that voters have policy preferences, and that they evaluate the distance between their own preferences and those of electoral candidates, and assess the candidates according to that measure. If you have an election in which a large portion of voters who prefer nationalisation, labour market protections and investment in social housing knowingly go out and elect candidates who want to privatise the NHS, deregulate labour markets and leave housing entirely in the private sector; well, something is up.

Specifically, what’s up is valence issues. You can broadly divide the issues of concern to voters in elections into two categories. The first category is position issues – these are issues on which parties, and voters, have divided views. Things like immigration policy, Brexit, nationalisation, labour market reforms and so on are position issues, because different voters and parties have different positions on these issues. Even where a majority of voters lean a certain direction (for example, about 80% of UK voters oppose a repeal of the fox-hunting ban), the existence of a minority who believe otherwise turns this into a position issue. We pay a lot of attention to position issues, because they fit neatly with a lot of fundamental theories about policy preferences. Perhaps more importantly, they also fit comfortably with most peoples’ basic understanding of how democracy is meant to work, and provide points of disagreement and debate which are interesting to follow as they unfold in newspapers and other media.

The second category of issue is valence issues. Valence issues are things on which the vast majority of people and parties actually agree. For example, “enhanced prosperity”, or “lower crime”, or “better education”, or “lower unemployment”; these are all things that just about every voter, and every political party, agrees to be positive. There’s lots of disagreement about how you achieve those things, of course, but fundamentally if you’re talking about issues of economic growth, human security and so on, you’re talking about a valence issue – something everyone wants to attain, regardless of where they fall on the political spectrum or how they feel about all the various position issues.

Jeremy Corbyn isn’t going to lose this election over position issues. On position issues, he’s good; the British electorate agrees with him, so much so that in an election where only the position issues mattered, he’d likely win the biggest majority Labour has ever held. You can imagine this in the form of a thought experiment; imagine a voting system where party and candidate names never appeared, and people simply selected their preferred policies, with their vote ultimately going to the party whose policies most closely match the voter’s. Assuming a kind of “veil of ignorance”, wherein voters could not guess which policies belonged to which party and thus couldn’t bias their selections according to party identification, Labour would win a huge majority this time out.

But Jeremy Corbyn is going to lose, because this election – like many in recent years – isn’t about position issues, it’s about valence issues. What valence issues boil down to is a simple question; given a core value that everyone agrees about, like “prosperity” or “security”, do you trust a given party or candidate to be able to deliver it? It’s not an assessment of policy, or a weighing of manifesto promises; it’s a simple, visceral and quite emotional choice of whether you think a person or a party has the competence to deliver the key social goods that a nation requires. Time and again in recent decades, we’ve seen electorates go to the polls, hold their noses, and vote for a party they fundamentally disagree with on many issues simply because they believe that that party is more competent and capable on the most fundamental issues of all, the valence issues.

Theresa May – for all that she has not been a particularly competent or capable leader, much as she was not particularly impressive as Home Secretary before – understands valence issues to a degree that Corbyn does not. While Corbyn has crafted policies on position issues which most of the UK electorate agrees with, May has focused entirely on projecting an image of strength and competence. She may be mocked for her constant and rather robotic delivery of her “strong and stable government” line, but it’s a good line; it speaks directly to the heart of the valence issues most people are basing their choices on. In fact, it’s rather hard to pin down the Conservatives’ exact policy positions on many things in this election, precisely because the whole party is running on valence. They’re avoiding talking about position issues, partially because they remain a party deeply divided on many of them, but mostly because their entire election pitch is that they’re a safe, competent pair of hands on the wheel, with little reference to where they’re actually planning on steering. Look also at how right-wing media and politicians alike respond to Labour’s policies. Rather than presenting an alternative or a competing worldview, their attacks are always based on claims that Labour is being unrealistic, or living in a fantasy land; that no matter how much you may like Labour’s policies (because the right wing knows that Labour’s positions are more popular), Labour in general and Corbyn specifically are too incompetent, too chaotic and too risky to put into power.

That’s why the Labour manifesto, for all that it’s a great document, isn’t going to mean much of anything in the long run. The problem isn’t that it’s too left-wing or too radical; it’s pretty apparent that the British public is quite receptive to some radical policy prescriptions on key areas right now. Rather, the problem is that Labour under Corbyn has done little to make people feel like the party has the competence to execute those policies. While those of us following the Brexit negotiations closely may be dumbfounded by the lack of competence and professionalism being demonstrated by the Conservative leadership in this area, that’s not the story that’s filtering through to the majority of UK voters. For them, the Conservatives are a competent party with some distasteful policies – and they’ll vote for that over a chaotic, incompetent party with lovely policies any day.

How did Labour get here? The blame, ultimately, has to rest with Corbyn; he’s leader, and the buck stops there. Certainly, the failure of the party’s centrists to unite behind the leader (even after their coup attempt collapsed) is also a major factor, but if Corbyn had cultivated a personal popularity beyond core leftist support then even his ideological opponents would have fallen in line. The party is fractured not because Corbyn has a different ideology to many of the Blair-era MPs, but because Corbyn is an electoral liability to the party. His great failure, I think, is that he truly believes that politics is about putting out the right policies and creating a manifesto people agree with; he has neglected the actual role of a modern party leader, which involves building a personal image of competence and leadership, and being an electoral asset for your party members around the country.

You can blame the media’s coverage of Corbyn and Labour for that negative image, as many of the party faithful do, and there’s some merit to that; but in the age of SNS and new media, Corbyn has shown no aptitude for engaging with the public through alternative channels and effectively challenging the narratives of the right-wing press. Again, I think, the problem is that he wants to let his policies do the talking, not realising that most people will not cast their vote based on policies. That’s a miscalculation that’s likely going to cost Labour a great many seats next month – because the greatest manifesto in the world is meaningless if you don’t believe Jeremy Corbyn is capable of delivering on its promises.

Tidying up Japanese SNS data for Machine Learning

In my last post about performing text analysis of Japanese language texts, I outlined how to install and use the MeCab system to break up Japanese sentences into their component parts (“tokens”) which can then be used for analysis. At the end of the post, I mentioned that if you’re using text sourced from social media like Twitter or Facebook, you might want to pre-process your data to deal with things like usernames, hashtags and URLs, which MeCab doesn’t understand or handle reliably.

Since then, I’ve spent some time building a tokenisation system to deal with very large volumes of data – the databases for the research project I’m working on at present sum up to about 15 million Japanese language tweets, and we expect to end up dealing with many times that volume by the end of the project. What have I learned? Quite a few things (not least a lot of stuff about using Google Cloud Platform, since this all rapidly outgrew my humble laptop), but one of the main ones was this; Twitter data is a god-damned mess. It’s not just hashtags, URLs and so on; people also tweet a lot of Emojis, which aren’t handled very well by a lot of analysis systems, and in Japan there’s also a tendency to tweet lots of “kaomoji” (you know, stuff that looks like “(。ŏ﹏ŏ)” or “((;,;;  ิ;;◞౪◟;; ิ;))“, and no, I don’t really know what that second one is meant to convey either…) as well as expressing feelings with single characters in brackets, like (笑) meaning laughter, or (涙) implying crying, which can also end up confusing the tagger.

A lot of conventional approaches to machine learning and text analysis just throw out those elements of the data. A common approach is to strip all punctuation, since it isn’t considered to have semantic meaning that’s useful to the machine learning system – but a kaomoji or a bracketed character clearly does have semantic meaning. The inclusion of a laughing kaomoji, or an emoji with a sweatdrop, or a bracketed character for crying, can radically alter the sentiment and meaning expressed by a tweet – in fact, they’re especially important on Twitter, where the 140 character limit means that people seek to find “economical” ways of expressing complex emotions and thoughts.

As a result, the tokenisation system I’ve built for our research is a fair bit more complex than I’d originally intended; it now strips out usernames, hashtags, URLs, emoji, kaomoji and bracketed characters from the data before passing it to MeCab to tokenise the remaining Japanese. There’s also a post-processing stage where I make sure that the keywords we used to build the data set (i.e. the Twitter search terms, which should appear in every single tweet in the data) are being tokenised as a single word, and not split up into separate words, as this could mess up analysis results further down the line. For the benefit of anyone trying to build something similar, this post will introduce all the systems I pass the tweets through in order to achieve this processing, in the order in which they’re done.


Finding Emoji in Tweets

Emoji – be they smiley faces, grinning turds or tiny depictions of churches and love hotels – have become ubiquitous on the Internet, but they turn out to be rather difficult to handle in text mining / machine learning approaches. In fact, some systems which don’t handle Emoji properly can end up making serious errors, as they not only misinterpret the emoji itself, but allow it to “pollute” their understanding of surrounding text characters too. MeCab doesn’t do a terrible job with Emoji, but frequently misinterprets them – so let’s find them and strip them out before passing the tweet over.

The problem is that this is a much harder task than it looks, because the standard for Emoji changes rapidly and isn’t simple. A certain number of ranges of characters in the Unicode standard (which is a system designed to create a standardised list of every character in every world language, thus ending garbled foreign language characters (文字化け) for good) are defined as being emoji – but the list isn’t fully agreed and is often updated. The most recent list I could find is from late 2016, and I’ve uploaded a copy of it here – feel free to download it and use it in your own project. The format it’s in is a Regular Expression (kind of a programming mini-language that allows you to do complex matching of characters and strings based on a set of conditions), and the way to use it in your Python program is as follows:

with open('emoji_list.txt') as emojifile:
    emoji_regex = ''.join(emojifile.readlines()).strip()
emoji_finder = re.compile(emoji_regex)

Now you can use emoji_finder as follows:

some_emoji = emoji_finder.findall(a_tweet)

This will return a list of the emoji in the tweet. I suggest adding them to the master list of tokens, and deleting them from the tweet itself before moving on to the next step of the tokenisation process. This is what you’ll do at every step; adding the elements you’ve extracted to the token list (along with a tag to say what kind of element they are, similar to the Part of Speech (POS) tagging that MeCab provides), and removing them from the tweet itself. Here’s a bit of sample code that does that:

for an_emoji in find_emoji(a_tweet):
    some_tokens.append((an_emoji, 'EMOJI'))
    a_tweet = a_tweet.replace(an_emoji, ' ')

Note that we’re replacing the emoji with a blank space, not deleting them entirely. This is deliberate; if the emoji was separating two words / sentences, i.e. the user was using it in place of punctuation, then shoving them back together could confuse MeCab and cause inaccurate tokenisation. If you’re building your own tokeniser, you’ll create a variation of the above function for every step along the way, so I won’t repeat the code for each one.


Finding Usernames and Hashtags in Tweets

Now that we’ve stripped out the Emoji, we can handle the tasks dealing with “ordinary” unicode characters. First let’s do the easy ones – usernames (which begin with @ on Twitter) and hashtags (which begin with #).

some_usernames = re.findall("@([a-z0-9_]+)", a_tweet, re.I)

some_hashtags = re.findall("#([a-z0-9_]+)", a_tweet, re.I)

Again, each of these functions returns a list. Note that they strip off the @ and # marks, so you should add those back in when you’re using a_tweet.replace() to get rid of them from your tweet text.


Finding URLs in Tweets

URLs have a number of consistent features, but they come in all sorts of shapes and sizes, and we need a system that effective matches all of those and pulls them out of the tweet. The below code is a Python adaptation of a regular expression originally created by John Gruber, which is designed to match any kind of URL, and seems to do the job very effectively – I haven’t yet found any URLs it doesn’t match.

Don’t worry too much about what the regular expression actually does; this is very much one of those cases where there’s no shame in copy and pasting a complex piece of code that’s well tested but which you don’t fully understand… (Incidentally, though I’ve put the two commands together here, you should actually create your “url_finder” object at the outset and re-use it over and over again for every tweet, instead of running the re.compile() command each time.)

url_finder = re.compile(r'(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'".,<>?\xab\xbb\u201c\u201d\u2018\u2019]))')

some_urls = url_finder.findall(a_tweet)

Finding Bracketed Characters in a Tweet

Next up, we’ll locate all those bracketed characters like (爆) and (汗) that often pop up in Japanese tweets (I don’t know if these have a name?). These are sort-of like a special case of the Kaomoji we’ll discuss in a moment, so be sure to strip them out before you do the Kaomoji – as mentioned before, these processing steps are being presented in order, and doing them in a different order may create some odd results.

bracket_finder = re.compile(r'[\((]' + re_text + r'[\))]')

some_brackets = bracket_finder.findall(a_tweet)

This will find the characters complete with their brackets, and will only work for single kanji (so it wouldn’t detect (爆笑) for example; but that’s a rare usage, and you start running into weird stuff like confusing this kind of character for people’s names being put in brackets etc.).


Finding Kaomoji in a Tweet

This is the really hard one. There’s no set standard for Kaomoji, and new ones seem to be invented almost every day. I really struggled to come up with a way to tokenise these peculiar beasts, until I came across a research paper from 2015 by a pair of researchers at Meiji University, Kurosaki Yuta and Takagi Tomohiro. They wanted to conduct a sentiment analysis test on Kaomoji, which is interesting in itself, but the part of their research that was really useful to me is the regular expression they constructed for locating the Kaomoji in text. Below is my Python version of their regular expression.

re_text = '[0-9A-Za-zぁ-ヶ一-龠]'
re_nontext = '[^0-9A-Za-zぁ-ヶ一-龠]'
re_allowtext = '[ovっつ゜ニノ三二]'
re_hwkana = '[ヲ-゚]'
re_openbracket = r'[\(∩꒰(]'
re_closebracket = r'[\)∩꒱)]'
re_aroundface = '(?:' + re_nontext + '|' + re_allowtext + ')*'
re_face = '(?!(?:' + re_text + '|' + re_hwkana + '){3,}).{3,}'
kao_finder = re.compile(re_aroundface + re_openbracket + re_face + re_closebracket + re_aroundface)

some_kaomoji = kao_finder.findall(a_tweet)

This works really well, except for one thing; it doesn’t know how to handle tweets with more than one Kaomoji present, so if you have a tweet like 「おはようございます!b(⌒o⌒)d 今日もいい天気じゃん!ヾ(〃^▽^)ノ」, it will match the outside edges of the Kaomoji and then extract everything between them – so we get a list with a single entry like this: ['b(⌒o⌒)d 今日もいい天気じゃん!ヾ(〃^▽^)ノ'], rather than what we actually want, which is both Kaomoji separately: ['b(⌒o⌒)d', 'ヾ(〃^▽^)ノ'].

My solution to this is not terribly elegant, but it is effective in every case I’ve tried thus far; I wrote a function that recursively divides up the string into smaller and smaller elements, and checks to see if there’s an individual Kaomoji lurking in them. Here’s what it looks like, with a sample call to the function at the bottom:

def kaomoji_find(a_tweet, facelist=None):
    if facelist is None: facelist = []
    faces = kao_finder.findall(teststring)
    for kao in faces:
        if len(kao) > 10:
            if len(re.findall(re_text, kao)) > 4:
                firstthird = kao_finder.match(kao[:int(len(kao) / 3)])
                if firstthird is not None:
                    facelist.append(firstthird.group())
                    facelist = kaomoji_find(teststring.replace(firstthird.group(), ''), facelist)
                else:
                    firsthalf = kao_finder.match(kao[:int(len(kao) / 2)])
                    if firsthalf is not None:
                        facelist.append(firsthalf.group())
                        facelist = kaomoji_find(teststring.replace(firsthalf.group(), ''), facelist)
            else:
                facelist.append(kao)
        else:
            facelist.append(kao)
    return facelist

some_kaomoji = kaomoji_find(a_tweet)

Keeping Project Keywords Whole in Tokenisation Output

Once you’ve done all the above steps, you’re ready to feed the remainder of the tweet to MeCab for tokenisation, just like we did before; then you can stick the MeCab tokens and the tokens collected in the above steps all together to create the “bag of words” for this tweet. Remember, the order of the words doesn’t actually matter to Bag of Words approaches to machine learning, so it doesn’t matter how we stick the lists of tokens together.

There’s one final wrinkle to deal with, though. If you’ve compiled your data set by searching for a certain keyword, that keyword or phrase will appear in every tweet – and you’ll want to be sure that it’s always tokenised in the same way. You don’t want MeCab splitting up your keyword in unpredictable ways, because this can mess with various kinds of analysis that you might be doing with your bag of words further down the line.

One approach to fixing this would be to treat keywords like we treated emoji and kaomoji – stripping them out of the text before passing them to MeCab. Don’t do this! Tokenising algorithms like MeCab use the surrounding characters in text to figure out where word boundaries are most likely to be; they rely heavily on the context in which a word appears to figure out which word it is and even where the word begins and ends. By taking out your keyword, you’re mangling up a sentence and preventing MeCab from tokenising it correctly (and honestly, if you’re using Twitter data, there are going to be enough mangled, ungrammatical sentences in there to make MeCab have whole baskets of kittens anyway).

The solution, then, is to let MeCab tokenise the tweet with the keywords still intact, then check through its tokens to see if it’s split the keyword(s) up anywhere, and replace them with a complete, un-split version if so. Again, apologies for my messy Python, but here’s the function I created to accomplish that:

def find_tokens(tweet, keywords=None):
    if keywords is None: keywords = []
    mt = MeCab.Tagger("-d /usr/lib/mecab/dic/mecab-ipadic-neologd")
    mt.parse('')   # Who knows why this is required but it seems to fix UnicodeDecodeError appearing randomly.
    parsed = mt.parseToNode(tweet)
    components = []
    while parsed:
        if parsed.surface != '' and parsed.feature.split(',')[0] != "記号":
            components.append((parsed.surface, parsed.feature.split(',')[0]))
        parsed = parsed.next
    for a_keyword in keywords:
        cindex = 0
        while True:
            if cindex >= len(components):
                break
            temp_key = a_keyword
            if components[cindex][0] == temp_key:      # If the keyword is already tagged as one item, no problem.
                cindex += 1
                continue
            elif components[cindex][0] == temp_key[:len(components[cindex][0])]:  # We just matched the start of a keyword.
                match = False
                tempindex = cindex
                temp_key = temp_key.replace(components[tempindex][0], '', 1)
                while True:
                    tempindex += 1
                    if tempindex >= len(components): 
                        break
                    else:               # Test next element.
                        if components[tempindex][0] == temp_key[:len(components[tempindex][0])]:  
                            temp_key = temp_key.replace(components[tempindex][0], '', 1)
                            if temp_key == '':
                                match = True
                                break
                            else:
                                continue
                        else:
                            break
                if match:
                    components[cindex] = (a_keyword, 'PROJECT_KEYWORD')
                    del components[cindex+1:tempindex+1]      
                cindex += 1
                continue
            else:
                cindex += 1     # This component doesn't match the start of a keyword, so continue.
                continue

    return components

A few notes on the above code. Firstly, it works like the other functions in this post – pass it a tweet, and it’ll pass you back a list of tokens – but it also allows you to optionally give it a list of keywords which it will check through and make sure they’re tokenised as a single item.

This version of the code also throws out all punctuation and whitespace (that’s the parsed.feature.split(',')[0] != "記号" part). I figured since we’ve extracted kaomoji etc., we can live without the remaining punctuation – it’s unlikely to be of value to analysis. If you have a different set of circumstances or requirements, you can remove that part of the code to hang on to punctuation tokens. Finally, this code doesn’t just output a set of tokens, it outputs a list of tuples in the form (a_token, part_of_speech) – with the part_of_speech bit being something like 名詞 or 動詞, indicating what kind of word MeCab reckons this is. For some analysis tasks, it can be useful to do something like excluding particles (助詞) or auxiliary verbs (助動詞) – again, this really depends what you’re trying to learn from your text.


Next Steps

And that’s it! Combined with the MeCab instructions in the previous post, that’s pretty much the set of components you need to build a pretty effective bag of words representation of a Japanese language social media post. It’s obviously a lot more complex than tokenisation for a “traditional” piece of text like a newspaper article, simply because people use text in unusual and non-traditional ways on social media. At some point I intend to do a test to see whether there’s a major difference in, for example, the sentiment analysis results you get from using a normal bag of words and my improved, pre-processed bag of words; I suspect there should be a measurable difference because we’re saving a number of elements with significant relevance to sentiment, such as kaomoji, that would be thrown out by a traditional bag-of-words processor. I’ll have to get the rest of our tool pipeline up and running before I can run a side-by-side test, though.

(Next post in this intermittent series will likely be something about how we end up processing and learning from our bag of words, introducing some of the core Python tools for natural language processing and machine learning such as scikit-learn and Gensim.)

Tokyo’s Tough New Governor is Picking all the Right Battles

My first piece for Japan Forward, the new English language site launched by the Sankei Shimbun earlier this week, has been published. It’s about Koike Yuriko, the new governor of Tokyo, and the uncompromising stances she’s been taking against corruption and crony politics in the large projects she inherited from her predecessors.

Japan Forward is a little rough around the edges for the moment – the site is officially still in beta and is certainly still finding its feet both in terms of design and functionality, and in terms of content – but the hope is that it’ll become a good forum for balanced, informed discussion of contemporary Japanese politics, policy and society. The Sankei Shimbun has a (deserved) reputation as a nationalist paper, but the staff involved with Japan Forward seem committed to developing a nuanced, credible and diverse publication with a distinctive identity, not “Sankei English Edition”.

I don’t agree with the Sankei’s editorial line on a whole lot, but one view we do share is that English language coverage of Japan is something of a monotone echo chamber – a relatively small number of like-minded people (some of whom I respect greatly, I should say) tend to quote one another around and around in a narrow circle that excludes other views and tends, as such echo chambers do, to spiral in towards some quite hardline viewpoints. “Abe is a Fascist” is a fine poster to wave at a street demonstration; as an editorial line in a respected publication, it’s not only questionable, it’s arguably irresponsible at a time when other countries are electing, or appointing, genuine fascists to positions of power.

My hope, though I am merely a contributor and not on the site’s staff, is that Japan Forward will provide a counterpoint to that kind of coverage – not by trotting out equally extreme viewpoints from the other side of the spectrum, but by highlighting, explaining and signal-boosting the centrist, mainstream political discourse of Japan that’s often lost in English-language coverage.

Assuming all goes well with the site, it probably means I’ll be posting a little less political material on this blog (and probably a bit more technical information related to machine learning and natural language processing, which are core foci of my doctoral research at the moment). For those following through RSS, though, I’ll be sure to post links to any pieces that are published on Japan Forward as they appear.

Japanese Text Analysis in Python

This is a more technical post than I usually write, but it will be useful to some people. My political science research involves some natural language processing and machine learning, which I use to analyse texts from Japanese newspapers and social media – so one of the challenges is teaching a computer to “read” Japanese. Luckily, there are some tools out there which make this (relatively) straightforward.

For this guide, I’m using Python 3.5. My development system runs on macOS 10.12 and I deploy my code to a server running Ubuntu 12.16, so this guide will include commands for setting up the software on both macOS and Linux/Ubuntu. I don’t use any version of Windows so I don’t know how to set this up on Windows – if anyone can provide a set of Windows commands that achieve the same goal, drop me an email and I’ll add them to the blog post.

What are we trying to accomplish?

The first hurdle to doing any analysis of Japanese text is segmentation, or tokenisation; breaking it down into usable chunks. For European languages, words have spaces between them, so you can just divide everything up at the word boundaries (the spaces, commas, full stops and so on), which yields an array of words that we can use to calculate things like frequencies or co-occurrence matrices. Japanese, however, has no spaces in its text, so there’s an extra pre-processing step required before we can start using these text analysis approaches.

In essence, we want to turn a string like this…

"今日はいい天気ですね。遊びに行かない?新宿で祭りがある!"

… into an array like this…


["今日", "は", "いい", "天気", "です", "ね", "遊び", "に", "行か", "ない", "新宿", "で", "祭り", "が", "ある"]

… which a computer can process to figure out frequencies, co-occurrences and so on.

A Software Shopping List

That’s the objective. Here’s a quick list of what we’re going to use to get there. (The URLs are for completeness only; we’ll be downloading everything from the command line.)

System Software:
MeCab

Dictionaries:
MeCab-ipadic
MeCab-ipadic-neologd

Python Libraries:
mecab-python3

This is not a definitive list of every piece of software that can segment Japanese text. There’s a big range out there, from the lightweight tinysegmenter through to fully featured software like kuromoji. I’m using this setup for two reasons. First, in my experience it’s the best at handling text data sourced from the Internet. Second, I think it’s the best trade-off of simplicity versus accuracy. tinysegmenter is much easier to set up and use, but its output is unreliable; it often breaks apart words that are actually common phrases or proper names (the name of the present Japanese prime minister, 阿部晋三, is rendered by tinysegmenter as [“阿部”, “晋”, “三”], not the correct [“阿部”, “晋三”] or [“安倍晋三”]; the same problem occurs with the word for prime minister itself, 総理大臣, which comes out as [“総理”, “大臣”] when what you (probably) want is [“総理大臣”]). Mecab works nicely with Python, and it’s easy to set it up with an extensive dictionary of common phrases and neologisms so its output is very accurate.

One other thing; if you’re following this tutorial on macOS, I expect you to have a little bit of familiarity with the Terminal. If you don’t know your way around Terminal at all, your homework is to go and install the “Homebrew” package; we’ll be using it to install a lot of the rest of the software. There are detailed instructions on the homepage and once you’ve done that, you’ll be able to get cracking with installing MeCab and its dictionaries.

Installing MeCab

First, let’s get MeCab (the core segmentation software) and MeCab-ipadic up and running. (Again, for macOS users, this assumes that you have successfully installed the Homebrew package.)

macOS:

brew install mecab
brew install mecab-ipadic

Ubuntu:

sudo apt-get install mecab mecab-ipadic libmecab-dev
sudo apt-get install mecab-ipadic-utf8

That’ll take a while, but once it’s done, MeCab should be up and running. You can test it by typing “mecab” at the terminal; in the blank line it gives you afterwards, type some Japanese text, and press enter. The result should look like this:

[email protected]:~$ mecab
無事にインストール出来ました !
無事 名詞,形容動詞語幹,*,*,*,*,無事,ブジ,ブジ
に 助詞,副詞化,*,*,*,*,に,ニ,ニ
インストール 名詞,一般,*,*,*,*,インストール,インストール,インストール
出来 動詞,自立,*,*,一段,連用形,出来る,デキ,デキ
まし 助動詞,*,*,*,特殊・マス,連用形,ます,マシ,マシ
た 助動詞,*,*,*,特殊・タ,基本形,た,タ,タ
! 記号,一般,*,*,*,*,!,!,!
EOS

As you can see, it’s working nicely and correctly identifying parts of speech in this test sentence. Press Ctrl-C to get back to the terminal command line, and let’s continue.

The next step is installing mecab-ipadic-neologd working. This is a bit more complex, since we need to download the dictionary of neologisms and slang (vital for handling text from the Internet) and then recompile it for MeCab. First, we  install the tools used to download (“clone”) the most recent version of the dictionary, then we compile and install the dictionary itself.

macOS:

brew install git curl xz
git clone --depth 1 https://github.com/neologd/mecab-ipadic-neologd.git
cd mecab-ipadic-neologd
./bin/install-mecab-ipadic-neologd -n

Ubuntu:

sudo apt-get install git curl
git clone --depth 1 https://github.com/neologd/mecab-ipadic-neologd.git
cd mecab-ipadic-neologd
sudo ./bin/install-mecab-ipadic-neologd -n

You’ll need to type “yes” at some point in the install process to confirm that you’re okay with overwriting some dictionary defaults. Now let’s check that it worked. To use this dictionary at the command line, we need to specify it when we invoke MeCab:

macOS:

mecab -d /usr/local/lib/mecab/dic/mecab-ipadic-neologd/

Ubuntu:

mecab -d /usr/lib/mecab/dic/mecab-ipadic-neologd/

(Note the different paths – if you’re on Ubuntu you’ll also need to change the path given in the Python files below to match.)

If you can type some Japanese text and have it tokenised (as in the previous example), then everything is working.

Working with Python

The next step is to get MeCab talking to Python. Again, this tutorial assumes you’re using Python 3 (I’m on 3.5, but any recent version of Python should be fine); there is also a library for Python 2, but I haven’t used or installed it. If you’re an advanced Python user and are using a virtual environment setup for your packages, you should switch to that environment now. (All commands from here onwards should work the same on both macOS and Ubuntu.)

At the terminal, type:

pip3 install mecab-python3

(The command may be “pip“, not “pip3“, on your system, but a lot of systems use Python 2 internally for various things, and therefore rename the Python 3 tools with a “3” suffix to avoid clashes. Also, depending on your setup, you may need to “sudo” that command on Ubuntu.)

Now open your Python editor, whether that’s an IDE (I use PyCharm personally) or just a text window, and let’s see if this is working. Try the following code:

import MeCab
test = "今日はいい天気ですね。遊びに行かない?新宿で祭りがある!"
mt = MeCab.Tagger("-d /usr/local/lib/mecab/dic/mecab-ipadic-neologd")
print(mt.parse(test))

When you run this code, you should see the same output you got at the Terminal earlier. (Note that if you’re on Ubuntu, you’ll need to change the location of mecab-ipadic-neologd in that code to match the location we were using earlier; the same goes for subsequent code in this example.) Okay, so Python is now talking to MeCab, but how can we turn that output into something useful that we can use in our analysis?

import MeCab
test = "今日はいい天気ですね。遊びに行かない?新宿で祭りがある!"
mt = MeCab.Tagger("-d /usr/local/lib/mecab/dic/mecab-ipadic-neologd")

parsed = mt.parseToNode(test)
components = []
while parsed:
    components.append(parsed.surface)
    parsed = parsed.next

print(components)

This will give you the following output…


['', '今日', 'は', 'いい', '天気', 'です', 'ね', '。', '遊び', 'に', '行か', 'ない', '?', '新宿', 'で', '祭り', 'が', 'ある', '!', '']

… which is what we’ve been aiming for from the outset! The parseToNode function of the Tagger generates an object that you can iterate over to see each token in the text. The actual token is accessed as “.surface“; if you want to see details of the part-of-speech or pronunciation, you can access that as “.feature“.

You’ve now got a system that will take Japanese text input (like a tweet, a Facebook post, or a blog entry) and turn it into a list of tokens. You can then apply exactly the same techniques to those tokens that you would to (more easily segmented) European languages.

A Few Cautionary Words

Before turning this system loose on a large volume of tweets or other social media data, I would strongly suggest writing some code to clean up your input – in particular, code to strip out and separately handle URLs, image links and (on Twitter) @mentions. MeCab doesn’t handle these well – it breaks up URLs into little chunks which will throw off a lot of machine learning algorithms, for example.

One suggestion (and the one I use in my own research) is to strip non-text elements from the text before handing it over to MeCab, and then add things like URLs and images back into the dictionary that MeCab returns to you – either as the full URL (ideally the non-shortened version) and @name, or as a token referencing a table that you can look up later (“URL1248”, “USER2234”).

Also, if you’re planning on analysing hashtags, note that MeCab doesn’t understand those either (it’ll tokenise the # and the text separately), so you may also want to pre-process those.

Finally, it’s quite likely that you’ll want to strip some elements out of the list MeCab returns to you as well. It has a habit of returning empty elements at the beginning and end of text, which you can remove; depending on the analysis you’re conducting, you may also wish to strip out punctuation, or even certain parts of speech (in which case you should check the “.feature” segment of each node to decide whether to keep or dispose of each given token).

Trump is no gift for Abe and the LDP

As Japan’s establishment and observers start to come to grips with the implications of a Trump administration for the country, one comment I’ve heard a lot is that this is a gift for Shinzo Abe and his inner circle. It clears the way for them to enact their long dreamed-of reforms, which would upend the post-war national order and return the country to a normalised military posture. Certainly, the almost certain cancellation of TPP is a blow to Abenomics, but in this view of events, Trump’s foreign policy stance is a huge boost for Abe’s militaristic agenda.

In one respect, that’s right. Trump’s election and the huge question mark it places over the stability and reliability of the 64 year old US-Japan security alliance does make Japan’s military normalisation vastly more probable. Even if the Trump administration does not, as seems very likely, demand an enormous increase in Japan’s financial contribution to the US military presence here (notably, Japan already pays a very large proportion of the US’ base costs) and talk down the nature of its commitment to the alliance, the unprecedented uncertainty his presidency introduces to the alliance will sway opinon among Japan’s political establishment and public alike. More significant military budget increases (not the small, conservative increases seen thus far) are almost inevitable; reform of Article 9 of the constitution, which looked all but impossible last week, may now be within the LDP’s grasp.

To call this a “gift” to the Abe administration, however, is a misreading of the reality. The Japanese government is not acting like the recipient of a long-desired gift; rather, it’s scrambling desperately to shore up and guarantee a continuation of the status quo. Abe was among the first leaders to speak to President-Elect Trump after his victory, not because he likes Trump’s positions but rather because Trump’s statements on Japan imply an uncertain and hence dangerous future for the East Asian order. Abe will also be one of the first world leaders to visit Trump and speak face to face with him in New York. In short, Abe and his officials are pulling out every stop and calling in every favour to ensure that this “gift” doesn’t actually materialise.

Why? Simply put, the LDP’s pursuit of remilitarisation and Article 9 reform has been the political equivalent of a dog chasing cars. It’s very fun and you get to make a whole lot of noise, but what on earth is a dog to do if one day it actually catches a car? It’s big, scary, hard and intimidating; the dog hasn’t the faintest idea what to do with it, and for all its chasing and barking, has never actually sat back on its hindquarters to come up with a plan of action for after the car is caught.

The LDP is a dog that just caught a car. Remilitarisation and the upending of the postwar order in Japan has been an issue that the LDP, and especially its more regressive wing (including Abe and much of his inner circle) has enjoyed harping on about for decades. Seriously, for all the talk of Japan’s right-wing shift or Abe’s nationalism, there’s nothing new under the rising sun; LDP leaders as far back as Nakasone in the early 1980s constantly banged on the same populist drum, and it was Koizumi back in 2001, not Abe, who reignited the whole mess around Yasukuni Shrine. Talking up military normalisation, making small, gradual changes (a peacekeeping operation here, a reinterpretation of legal advice there) to the existing order and muttering loudly about “masochistic” accounts of history or “correcting” other countries’ viewpoints is red meat for a certain portion of the LDP base. It’s a strand of populist rhetoric that has been a part of the LDP’s messaging since Nakasone, but in a relatively low-key way; the LDP knows that most of the Japanese electorate supports the constitution, dislikes the idea of overseas military engagements, rather likes the postwar order and will only tolerate this kind of populist rhetoric as long as it’s seen, broadly, as harmless letting off of steam by slightly daft nationalists.

The result is that for all the talk, all the noise and the chest-beating, there is no plan in place for Japan to take independent control of its national security. There is no plan for full normalisation of the Japan Self-Defence Forces into an actual national military. There is no roadmap for any of this. There isn’t even a coherent plan for reforming the constitution; the much-vaunted “ideal constitution” drafted by LDP right-wingers and posted online some years ago is a fever-dream of return to a dimly imagined glorious past that none of its authors ever imagined actually putting into practice. As for the JSDF, it is an impressive military force in its own right – extremely well-trained, well-organised and by far the most technologically advanced military force in Asia – but it’s designed to function in concert with the US military, fulfiling very specific roles alongside the much more capable and flexible US forces. Upgrading, repurposing and realigning the JSDF into a military capable of independent operation is an enormous undertaking – expensive, difficult and time-consuming.

That’s the car the LDP has been chasing, and has just caught. The most extreme (and thankfully unlikely) version of what happens next sees President Trump fulfil his most hardline electoral promise – vacating the US’ bases in Japan and pulling back US forces from East Asia generally. While the JSDF is a competent and well-equipped force, it is in no position to fill that vacuum; even on something as simple and vital as missile defence, Japan’s advanced AEGIS destroyer fleet relies upon the presence of US vessels to fill gaps in the shield through which a North Korean missile might slip, and the timescale for achieving operational independence in that regard alone is on the order of many years. A less extreme (and arguably the most likely) version sees the US remaining in Japan, but demanding more financial support for its presence, and appearing less robust in its commitment to defend Japan’s extremities – including sort-of contested (in a “China making up flat-out nonsense” sense that has become wearily familiar across Asia of late) territories like the Senkaku Islands. While that would reduce some pressure, ensuring that the US military would remain involved in Japan’s security, it will still demand essentially the same long-term response from the Japanese Govermment. If there is even the slightest doubt in the US’ willingness to defend Japan from attack, Japan has an absolute requirement to shift the posture of the JSDF – constitutionally, legally, technologically and strategically – to that of a normalised military capable of independent functioning.

Abe and his inner circle don’t want to do that. Sure, they’d like that to happen, but the passive voice is important here; it’d be great if someone else just went and did it overnight with a sprinkling of magical fairy dust, but the daunting, years-long project of turning around Japan’s strongly opposed public opinon, its constitution and legal system, its military stance and its governing institutions to allow such a change is a minefield the LDP never really planned on actually walking into.

That’s why Abe was so quick to lift the phone to President-Elect Trump, and why he’s been so quick to arrange to meet him in New York. The opportunity to reform Article 9 has never been greater, but that’s a distant second in Japan’s priorities right now; its most pressing and urgent priority is to head off the disaster that would be the US backing off even slightly from its commitment to the US-Japan alliance. Constitutional reform and military realignment is on the table, but it’s no gift; nobody in the LDP with any clarity of outlook is smiling at the prospect right now. Maintaining the status quo as much as possible will be the number one priority of Abe, his administation and any of his possible successors for the duration of the Trump presidency.

What President Trump means for Japan

Donald Trump is, failing an electoral college miracle, going to be the next president of the United States of America. Countless words will be written about what that means for America, but as a resident of Japan and a scholar of Japanese politics, I’d like to talk a little about what a Trump presidency means for Japan.

This is a question whose implications extend far beyond Japan itself. While the UK (and Australia) make much of their “Special Relationship” with the USA, Japan has in many ways been America’s most steadfast and important ally in the postwar era. The security treaty between the two countries is a cornerstone of the geopolitical order and stability of East Asia; Japan’s development as an economic powerhouse, aided and abetted by the United States, created a bulwark against communism in Asia; its embrace of democratic values made it a template of Asian democracy in a century when that was often a rare commodity.

While Japan has generally viewed Republican administrations in the US as being more amenable to this relationship, the reality is that the US-Japan alliance – both the formal security alliance and the more complex mesh of economic and political arrangements that bonds the nations – has been supported and developed by both Republican and Democrat administrations since its origins in the late 1950s.

Donald Trump’s stated positions on foreign policy are a significant threat to that relationship. Trump’s statements throughout his campaign paint Japan not as a partner but as a global rival of the United States. He suspects Japan of currency manipulation to the detriment of the US, and explicitly opposes the Trans-Pacific Partnership free trade deal of which the US and Japan are the key players. He has also repeatedly railed against what he views as the United States being short-changed in its security arrangements with Japan, and has gone far beyond the US establishment – which has long pressured Japan to take a more proactive role in conflicts and peacekeeping efforts – by suggesting that Japan should arm itself with nuclear weapons to ensure its own security.

That platform blindsided the Japanese political establishment; it’s a blast from the almost-forgotten past of the 1980s, when Japan’s rapid development and aggressive overseas acquisitions spurred fears that it would overtake US economic dominance. However, it’s hard to say how much of the platform will actually make its way into US policy. TPP is almost certainly dead in the water, but what line the US will take on Japan’s supposed currency manipulation and on broader trade issues is unknown. The security alliance, meanwhile, is based upon a ratified international treaty which will limit the actions of even a US President with the House, Senate and Supreme Court all lined up behind him – but its implementation, and the scope of the security cooperation between Japan and the US, could certainly be influenced if the Trump administration takes the insular, isolationist approach to foreign affairs that seems most likely.

No matter what policy or structural changes ultimately result, however, a degree of damage has already been done. Alliances are not just built upon legal treaties; they also rely on relationships of trust and assurance – upon both parties believing, long-term, in the commitments made by the other. Moreover, in the case of security alliances in particular, it’s not just the parties to the alliance who have to believe in it; outside actors, looking on at the alliance, also need to believe that its commitments are firm and will be upheld if tested. If an outside actor feels that the alliance is a paper tiger – that its commitments would not be upheld under testing conditions, for whatever reason – it creates a flashpoint for conflict and a weakness in the structure that upholds regional or even global stability.

Trump’s election, after more than a year of anti-Japanese campaign rhetoric, will weaken both internal and external perceptions of the Japan-US alliance regardless of what actual policy changes result under his administration. The view of the USA as being an increasingly unwilling participant in the affairs of East Asia will only be enhanced by Trump’s similar reticence regarding America’s role in NATO in Eastern Europe; a narrative of US isolationism will take hold among geopolitical rivals who chafe at the existing order, including Russia and China.

That is a tremendously dangerous position for Japan to find itself in. From economic rivalry to unresolved wartime history issues, the country’s relationships with several of its neighbours are fractious – most notably with China, a burgeoning superpower with thinly veiled imperial ambitions that stokes anti-Japanese sentiment as a release valve for discontent among its populace, and with North Korea, a nuclear-armed failed state. This not a hypothetically dangerous position; in absolutely practical terms, Japanese planes and ships tangle with Chinese incursions into the country’s airspace and territorial waters on a daily basis, while North Korean missile launches become increasingly sophisticated and deadly with each passing month. The importance of the US in maintaining Japanese security in these regards is paramount. The US commitment to the defence of Japan deters China from escalating its confrontations into what it would expect to be a brief, glory-seeking conflict and seizure of Japanese islands. In an even more practical, nuts-and-bolts sense, the US’ AEGIS destroyers are an essential part of Japan’s missile shield, with the country’s own fleet, although more advanced than any comparable navy in East Asia, presently incapable of dealing with the latest generations of North Korea’s missiles without US support.

If China or North Korea view America’s commitment to Japan’s security as negotiable or softening, either party may attempt to test the waters in a way that could lead to a much broader and bloodier armed conflict. In anticipation of that, and reflecting Japan’s own dawning unease regarding America’s commitment to the alliance, it’s almost certain that the US establishment will now get its long-held wish, albeit in a way it never wanted or expected – with Japan pushing harder than ever to normalise and expand its military prowess in order to make up for perceived US weakness (or non-commitment) in the Asia-Pacific region.

Quite a few commentators on Japanese politics and policy would argue that this is a process which has already started, but as I’ve observed before on this blog, Japan’s military budget increases have actually been extremely limited in recent years, with the country continuing to treat the US alliance as the beating heart of its security arrangements. The possibility of a revision to the pacifist Article 9 of the postwar constitution, though desired by Prime Minister Abe and his inner circle, had looked very remote indeed – until Trump’s election. Now, it is guaranteed that America’s relationship with Japan and the depth of its commitment to Japan’s security would be a fiercely debated topic for the coming four years. Many moderates in Japan will likely conclude that while Japanese pacifism was wonderful while the country remained safe behind America’s shield, if that shield can no longer be fully relied upon (and if China and North Korea suspect that the shield is less impervious than it used to be) then Japan has an urgent, pragmatic need to arm itself, and to remove the legal restraints that might prevent its military from effectively defending the country.

It’s not just Japan’s security position and the likelihood of normalising its military role that will be heavily impacted by the Trump presidency. The Japanese government, assuming a Clinton victory, had sought to pressure the US on TPP by ratifying the deal this month, well ahead of the inauguration of any new president. With Trump taking the office, TPP is likely off the table, and with it goes one of the core pillars of Abenomics. How Japan will react is unknown, but it seems likely that the country will feel compelled to seek out alternative trade arrangements – a Plan B to shore up its troubled economic reform programme. A version of TPP negotiated between Japan and the other non-US signatories is one possibility. Closer ties with Russia are another, although Russia’s economy is something of a disaster and Japan’s bureaucrats may be worried about hitching their cart to that particular horse. A long-discussed Japan-EU deal might even be expanded, though for a full-spectrum deal, the EU wants Japan to look at things including abolishing its (grossly abusive and cruel) death penalty system, which would be a sticking point. None of these, though, would match the sheer volume of trade that would have been affected by the TPP’s liberalisation of the cross-Pacific trade between Japan and the US. Regardless of your view on TPP itself (personally, I think it’s a mess, with far too many self-interested parties involved in opaque negotiations that have ultimately yielded an over-complicated, ill-considered, under-researched and worryingly anti-democratic treaty – but it’s still probably better than the existing situation), this is a huge stumbling block for the plans for economic reform and recovery in Japan.

This is where we stand now, only hours after Trump was elected. We don’t know who his key appointments are, what his policies will be, or any other concrete detail – but when the USA sneezes, Japan catches a cold. Trump being President-Elect already has clear, powerful impacts on Japanese domestic and foreign policy. The country’s economic programme is facing a deep crisis. Meanwhile, the likelihood of “remilitarisation” (really, just a normalisation of Japan’s military to the same status as that of any other developed nation, but likely to stoke tensions in East Asia nonetheless) and constitutional reform just took a powerful shot in the arm. With Trump preparing to enter the Oval Office in January, Japan is for the first time since the 1950s being forced to consider that its future might not include a close US relationship – and that is, of necessity, going to yield a very different Japan.

Seeking the Roots of Trump’s Support

Donald Trump’s supporters are the people who have been left behind by successive waves of economic change. They are the people who in the space of a generation have seen solid, stable manufacturing jobs turn into short-term, unstable, low-paid service industry jobs – or worse, into unemployment. They are the people who lost out to free trade – whose jobs went to China and India, and who gained nothing from what came back in return. They are the people who fell through the cracks of the economic recovery Barack Obama has overseen since 2008, just as they fell through the cracks of every recovery since the 1980s, and that has taught them to distrust statistics in general and politicians in particular. They’re voting for Trump because the current political status quo has given them nothing and perhaps burning it down will make things better in the end.

Or perhaps;

Donald Trump’s supporters are the people who have been unable to accept or adapt to the cultural change around them. Around a hard-core of white nationalist racists is a mass of support drawn from people who have seen the nature of their communities change due to immigration, equal marriage and secularisation. They are people who don’t understand why TV needs to have so many gay characters, especially in shows children might be watching, or why young people listen to hip-hop, and who feel like the values they grew up with – respect for authority and a sense of white, Christian identity – are under constant attack from an urban, liberal elite. They’re voting for Trump because he stands up and says the things they believe in blunt ways that make the dog-whistling of other Republicans seem cowardly.

Which of those statements is true? That’s become one of the biggest questions in political science – not just with regard to Donald Trump, but with regard to the rise of radical populism, both on left and right, across Europe and much of the developed world. The economic explanation has more devotees. The cultural explanation has a growing body of evidence behind it. Parties supporting both camps tug back and forth.

Support for Trump in economically depressed, traditionally Democrat-leaning towns of the Rust Belt and the North East is presented as a knock-out blow for the Economics camp. The Culture camp counters by noting that the average income of a Trump supporter is actually above the national average for people of their demographic group; those backing Trump are not, by any means, the economically hardest-hit of the nation.

Neither camp is entirely convincing. The reason for that is simple; neither factor, taken in isolation, can adequately explain support for Donald Trump (or for Europe’s radical populist parties, or for Brexit). By arguing for the supremacy of one argument over another, political scientists ignore two important things.

Firstly, political support is always a coalition, not a hive mind. There is space within Trump’s support base – currently standing, according to the polls, at between 38 and 40 per cent of the US electorate, or around 90 million voters – for people motivated by economic factors, people motivated by cultural factors, people motivated by a mixture of both and even, undoubtedly, people motivated by other things entirely (like the suggested group of evangelical Christian voters who have reportedly convinced themselves that God’s plan is to put Trump in office and then have him die or resign so that the right-wing Christian VP candidate, Mike Pence, can take his place). There is no theoretical requirement for a single factor to be able to explain a majority, or even a significant plurality, of a candidate’s support.

Secondly, economic hardship and cultural intolerance are not independent of one another. One of the things you always have to look out for in any kind of political research is correlation between the factors you’re investigating. Treating two factors as being independent, when they actually interact with one another or with other variables in some important way, can seriously break the model you’re trying to build.

That’s almost certainly what’s happening here. It’s not as simple as “poor people are more intolerant than rich people” – that’s both a gross oversimplification, and provably untrue in much of the data available. However, there are lots of ways in which those factors may interact that are a little more complex and worthy of consideration.

Take for example the urban-rural divide. Urban communities in the USA are overwhelmingly voting Democrat; rural communities generally go Republican. That pattern is true in polling for this election too; one of the most effective predictors for whether a county will vote Trump or Clinton is its population density. In part, that’s because cities have larger minority populations than the countryside; but urban white people lean more towards Clinton, too.

There are cultural differences between urban whites and rural whites, not least of which is that urban whites live alongside minorities (ethnic, sexual, religious and otherwise) and that experience has been shown to push people towards being more tolerant and liberal. In the UK, the biggest electoral successes for xenophobic far-right parties like the BNP and UKIP came in towns with the smallest numbers of immigrants; familiarity with the Other, well, stops them being the Other. That’s definitely a factor in the Trump support equation.

There are also, however, economic differences between urban whites and rural whites. Urban whites are not necessarily better-off than rural whites (indeed, they’re often less well off), but for the most part the economic opportunity available to them is greater. They live in diverse cities where the end of one kind of industry usually means the springing up of another. Metropolises do die on occasion, but it’s rare; mostly a period of decay following the decline of an old industry is followed by new businesses moving in to take advantage of lowered costs. Rural areas, though? Small towns and villages? When they die, they die. One factory closing can turn a small community into a ghost town; people who have the mindset and the opportunity to move to a city do so, and what remains is a zombie village, lurching forward through inertia but likely doomed to decay and decline forever.

In short, a rural white voter may have cultural differences from his urban cousin, but he also has a key economic difference; even if he’s doing fine personally (though again other factors come into play – is he doing as well as his father did, or worse? Perceptions of economic conditions are highly influenced by expectations, after all), he may be surrounded by a community in decline, driving past boarded-up windows every day, constantly reminded that his income and lifestyle is fragile, and fearful that his country has forgotten his kind of town even exists.

See how that imaginary voter ends up being a melange of both economic and cultural factors? My hypothesis is this; Trump’s voters are generally motivated by a little from column A and a little from column B. For those who harbour racist, xenophobic, homophobic or otherwise regressive social sentiments, they were willing to just roll their eyes at the multicultural liberals in the cities and calmly vote for mainstream, non-populist candidates as long as their communities remained economically stable and prosperous. These towns may not have been the most welcoming places in the world for minorities, but for the most part minorities were not blamed for major problems (a core underlying theme of radical-right populism, and of the Trump campaign specifically) because there weren’t really any major problems to blame them for.

On the flip side of the coin, you’ve got those who have been severely impacted by the economic shake-up of the developed world, and left behind by globalisation and technological progress, but who have not reacted by supporting Trump. That’s because they didn’t have latent regressive sentiments that could be activated by economic hardship (either their own, or that of their community as a whole). They live alongside minorities, or perhaps have minority groups represented in their own family and close friends; they’re college-educated, which in addition to giving them a different context in which to consider and critique Trump and Clinton’s competing claims, has exposed them to many minority groups (even the most conservative colleges have far more opportunties to meet and get to know people unlike yourself than small town communities generally offer). Though times are tough for them, they were never in the potential Trump voting camp, because they don’t have a culturally grounded anger at minority groups which they can shape into blame.


There’s another group worth mentioning; the “always-Republican” group of people for whom voting Republican is part of their identity (though they’d blanch at the thought of actually subscribing to identity politics, oh dear me no), for whom Clinton is too left-wing – and despite her progressive detractors, she is notably more left-wing and progressive than even Barack Obama on almost everything except foreign policy. For this group, and there seem to be a hell of a lot of them, voting Trump is partially about identity, and partially about ideology. They’ll happily admit that the man is a boor, a misogynist, a racist, even a fool; but they worry about the country’s direction under Clinton, and have convinced themselves that Trump will be effectively restrained by the Republican Party and by the nation’s democratic checks and balances, making him into a far more moderate-right leader than any of his own statements and positions have implied.

These people skew the data. They’re not really “Trump supporters”; they’d vote for anyone wearing the GOP badge, or perhaps simply for anyone standing against Clinton. They share some of his views but dislike how he expresses them (the dog whistling was just fine for them). They’re the Republican hardcore, inseperable in the data from the true Trump supporters, and they mess up the statistics; they’re likely older, and richer, and perhaps less religious and more educated than the kind of people who actually wear Make America Great Again baseball caps.

One might also observe that pretty much this exact demographic has also served the function of useful idiots ushering in their reign of every right-wing authoritarian strongman in history – relatively moderate traditional conservatives who give the reins of power to a demagogue in the mistaken confidence that they’ll be able to control him once he’s in charge – but that would be unkind. True; but unkind.


The 90 million or so people who currently say they support Donald Trump (assuming 100% turnout, of course; the actual vote figure will be far lower) is an enormous, diverse demographic. It’s not necessarily diverse in a traditional sense – it’s overwhelmingly white and a bit of a sausage party, with lots of traditionally Republican-supporting women seemingly crossing the aisle this year – but the backgrounds, reasoning and motivations of those in that group are incredibly disparate, which makes a bit of a mockery of attempts to find the Holy Grail, the one, true reason why people are voting Trump.

Perhaps confusing matters even more are the core supporters and true believers – the 13 million people who voted for Trump in the primaries (though some of those early votes were likely half in jest, and some of the late ones made through clenched teeth to keep the arguably even more odious Ted Cruz from the nomination), for example, or the thousands who turn up to his rallies around the country. Their motivations are perhaps more clear, and certainly more clearly telegraphed. Those who picked Trump from among a stable of boring but broadly moderate Republicans, and those who turn up to cheer him on at rallies, have an agenda that’s clearly cultural. A large portion of them are absolutely cheering on the turning back of the clock, the disenfranchisement of ethnic and sexual minorities who have “skipped the queue” at the expense of “real” Americans; they may have been activated by economic hardship, but the cultural base of their grievances, and the racist White Nationalism to which it has led them are extremely clear.

There aren’t very many of these people, though. Even if you accept that every single person who voted for Trump in the Republican primary was doing so on the basis of cultural regressivism and white nationalism – and I think that’s absolutely a ceiling figure, not a realistic estimate – that’s only 13.3 million people, in a country with 225.8 million eligible voters. About 6% of the population voted for Trump in the primaries. His gains from that figure may partially reflect an anger at cultural change, or an anger at economic instability, or more likely a mixture of both, but a much larger component is simple partisanship; Republicans voting Republican because they’ve always voted Republican, and/or because they don’t like the Clintons.

6%. That’s the deep, devoted core of Trump support. A higher percentage than that say they’re going to vote for the bumbling, comedic Libertarian candidate, Gary Johnson, on November 8, and yet we’re not falling over ourselves to understand why Johnson supporters are the way they are, what has motivated them, and what has activated them. (In fact, we’re also not falling over ourselves to ask why a majority of so many demographic groups are voting for Clinton; it’s only Trump that provokes this fascination.) It’s not that understanding this isn’t worthwhile; I’m a political scientist precisely because I want to understand everything, even while recognising the futility of the quest. Rather, it’s the assumption that the things that have driven and motivated those people should somehow be taken into account, listened to sagely, nodded at in understanding, and allowed to influence the future direction of a nation or even a planet.

6%. That’s smaller – perhaps less than half – than the percentage of the population from whom the 6% would like to remove the right to marry and the right to a family. It’s far, far smaller than the percentage of the population from whom they would deny the right to equal treatment under the law, or to schemes designed to rebalance historical inequalities. White Nationalism truly is White Supremacy; it assumes that a small group of white, conservative and under-educated men should be allowed to dictate the fates of far larger groups of people. This is a dark fantasy that is only fed and watered by earnest hand-wringing over their motivations and reasoning.

Of course, improving the economic conditions of everyone – yes, even racists, I guess – should be the role of goverment; the decline of communities and the installation of a safety net and alternative path for those impacted by globalisation and technological progress should be a priority. It shouldn’t be a priority because of Trump’s support; it should be a priority because it’s the right thing to do. Perhaps it will head off future waves of populism; the literature on cultural backlash suggests otherwise, but it certainly can’t hurt, especially if there really is some correlation between cultural intolerance and economic instability.

But when anyone talks about changing policies or slowing down the rate of social progress in order to attune to the desires of the Trump supporter, remember who we’re actually talking about; the 6%. Everyone else is a johnny-come-lately, a fairweather Trumper, a half-hearted enthusiast for anyone wearing a GOP badge on their lapel, a self-deluding cheerleader for a demagogue they assume, despite all evidence to the contrary, to be on some kind of leash (don’t you think that if the Republicans could control Trump, they’d have, well, done it by now?).

Understanding Trump’s support – in terms of cultural identity, in terms of the plight of economically depressed communities, or in whatever other terms are found to make sense – is important. Don’t let anyone tell you that the next step after understanding it should be pandering to it.

Restraint, not Aggression, in Japan’s Military Budget Increase

JSDF troops with their flag

The remilitarisation of Japan is a popular theme for the international media. It gives a clear, dramatic narrative to international news coverage that might otherwise bore readers. In this narrative Japan’s leadership seek to cast off the shackles of the post-1945 world order, to rewrite the pacifist constitution, rebuild their military forces, inculcate hatred of their Asian neighbours, and adopt an aggressive, warlike stance towards Asia. Leading the charge is prime minister Shinzo Abe, with fellow members of the shadowy conservative/revisionist Nippon Kaigi organisation being given senior government positions from which to realise their militaristic goals.

Not all journalists or publications buy this narrative in its entirety – but either in full-blown “Abe is a Fascist!” form or in a more diluted manner, it has become the master narrative of Japanese politics in the international press. That narrative frame can be seen in coverage this week of the request by Japan’s Ministry of Defence for a 2.8% budget increase. “Japanese Government Urges Another Increase in Military Spending” reports the New York Times; “Japan Defence Ministry seeks Record Budget to Counter Chinese Threat” says The Guardian. Both stories, in common with most coverage of the budget request, emphasise that this is part of an ongoing process of (re)militarisation.

I don’t wish to single out the NYT or the Guardian, nor the writers of these articles (Mokoto Rich and Justin McCurry respectively) – my intention isn’t to bash their coverage, which is actually more even-handed and well-researched than a lot of other articles on this topic. Rather, I’m linking to those articles to demonstrate that even the better news outlets continue to support a narrative about Japan which deserves to be questioned more closely.

There are lots of questions to be asked about this narrative. We might ask why Nippon Kaigi, for all that some of its policy positions are unpleasant or ill-informed, is considered any more shady than other political lobbying groups. We might ask why an organisation portrayed in the media as a shadowy background powerbroker would have an extensive and informative website setting out its aims and policies, or media briefings with its leaders – including one fairly recently at the FCCJ, Tokyo’s foreign correspondents club. We might also ask why, following the recent House of Councillors’ election, media outlets almost universally reported that Abe’s government now had the votes necessary to reform the constitution, ignoring the fact that many of those who support constitutional reform (including the LDP’s coalition partner, Komeito) support entirely different proposed reforms to the LDP – not to mention that any reform would need to pass a referendum, too.

This week’s conversation is about military budget, though, so let’s look at military budget.

Graph of Military Budget in US$

Military Budget in 2014 US$

This graph shows the military budget of Japan and some of its neighbours over the past 20 years – from 1995 to 2015 – in millions of US$. For ease of comparison, all figures are normalised to 2014 US Dollars. Two things are immediately apparent.

First, Japan’s expenditure hasn’t changed much in 20 years. There were some large rises towards the end of the 1980s, not least because the United States demanded that Japan should pay more towards the cost of US bases on its soil, but since the mid-nineties Japanese expenditure has stayed fairly solid in US$ terms. In fact, its military budget is almost identical to that of Germany, and significantly lower than the United Kingdom – a smaller island nation in a significantly less turbulent part of the world.

Second, Japan’s neighbours are spending huge amounts on military expansion. China’s budget, three to four times greater than Japan’s and growing at 7 to 8% each year, is now second only to the United States (the US isn’t on this graph because it’s ridiculous – the scale required to show the US’ military spending, more than 10 times that of Japan’s, squashes all other countries into a multicoloured line bouncing along the bottom of the graph). Russia now spends double Japan’s budget. South Korea, with less than half the population but a more precarious defence situation, spends a comparable amount to Japan.

We have no data for North Korea, whose aggressive nuclear weapon and missile programs are one of the main reasons for Japan’s budget increases, much of which will be spent on improving missile defences.

Here’s a second way to look at the data.

Graph of Military Budget as a Percentage of GDP

Military Budget as a % of GDP

This graph shows the military budget of Japan, its neighbours and some other countries as a percentage of their GDP over the past 20 years. In some ways it’s a misleading chart – while China looks fairly flat on this graph, its GDP has boomed so the cash it spends on the military has grown enormously even without using a larger proportion of GDP. Japan, meanwhile, has had mostly stagnant GDP figures for the past couple of decades. With those caveats in mind, though, we can pick out some interesting things from this data.

We can see that Japan spends far, far less on defence as a percentage of GDP than pretty much any other major nation. Russia’s expenditure is off the chart (literally), while the USA, South Korea, the UK and China all spend over 2% of their entire GDP on the military. Japan doesn’t belong to the same category of nation at all; in fact, its GDP percentage spend is lower than Germany. The closest nation in the data set to Japan, by this measure, is the notoriously militaristic, sabre-rattling, neighbour-terrifying, aggressively warlike… Canada.

Incidentally, out of every single country in Asia and Oceania, only three spend less of their GDP on defence than Japan – Indonesia (0.9%), Mongolia (0.8%) and Papua New Guinea (0.6%). (If you’re interested, the lowest military budget as a percentage of GDP of any nation in the world was the 0.4% spent by Ireland, Guatemala and Nigeria. Famously neutral Switzerland spent 0.7%.)


None of this is to say that there aren’t some problematic things about Japan’s political trajectory. Abe and his close associates have troubling views on history, and there are valid fears that those views will drive his government towards policies which promote nationalism and xenophobia and erode international ties in East Asia. Much more worrying than anything about Nippon Kaigi is the extent to which his unprecedented dominance of the LDP has shut down intra-party competition and debate; the LDP used to be its own best opposition thanks to healthy competition between factions, which is now all but moribund. And yes, certainly, criticism is due of politicians (in all countries!) who can’t seem to control their childish urges to provoke their neighbours over historical or territorial disagreements.

The master narrative of Japan’s slide towards remilitarisation, nationalism and even fascism, however, just isn’t supported by the facts. Take constitutional revision; while being more seriously considered than at any point since the 1950s, it still has to clear many tall hurdles. More aggressive ideas for changing the constitution are not even supported unanimously by Abe’s LDP colleagues, let alone by the other parties whose support would be needed or the Japanese public who would vote in an eventual referendum.

The increases in the military budget, meanwhile, are eye-opening not as proof of militarism, but as proof of extraordinary restraint. Faced with enormous military build-ups in neighbouring nations – two of whom, North Korea and China, have carried out minor but overtly hostile actions towards Japan in recent years – Japan’s military spend remains modest. It has the third-largest economy in the world but spends less on its military than France or the UK – neither of whom experience either regular military / paramilitary incursions into their waters, as Japan does from China, or the testing of ballistic missiles aimed across their territory, as Japan does from North Korea. (Neither do the neighbours of France and the UK promote educational policies which distort historical fact to demonise France or the UK, tolerate rioters attacking outlets of French or British businesses, or broadcast endless TV shows dramatising often exaggerated accounts of French or British war crimes.)

In the face of these threats, and against a background of increasing pressure on the Japanese Self-Defence Forces to participate fully in international peacekeeping and reconstruction missions (Japan has been bashed for decades in the international community for sending cheques rather than physical assistance to stricken areas), the modest increases to Japan’s defence budgets suggest caution and restraint. Japan remains Asia’s most successful democracy and it relies for much of its security not on enormous military expenditure but on the strength of its diplomatic and economic ties around the region and the world. In the face of real concerns over regional stability in Asia, it would be helpful if the international press desisted from attempting to undermine that position for the sake of more dramatic headlines.

Yasukuni and the Politics of Remembrance

Yasukuni Shrine

Yasukuni Shrine is a place and a political controversy that features in a number of posts on this site. Many of the views you’ll read about the shrine are shrill and one-sided; I thought it might be useful, as a reference piece, to write up something more balanced about the shrine’s history and its present role in politics and society.

August 15th marks the anniversary of Japan’s surrender and the end of the Second World War. It’s an important and emotive date for many Japanese people. Many still alive today recall the events of 71 years ago. Countless others have memories of parents, siblings or friends lost to the war. The anniversary, by coincidence, falls during Japan’s Obon festival, during which the souls of one’s ancestors are worshipped, and graves and shrines visited.

In recent years, August 15th has taken on large and unfortunate significance for observers of Japanese politics and East Asian geopolitics. It’s become a barometer for the strength of Japan’s right-wing, revisionist political lobby, who argue for an end to the nation’s post-war order and to “masochistic” views of wartime history. Related to this, it is a barometer for the state of the relationships between Japan and its nearest neighbours, South Korea and China.

At the heart of that significance sits Yasukuni Shrine. Prime Minister Koizumi Junichiro lit a match under the shrine’s political role when, in 2002, he pledged to make official visits to the shrine each year. The power of that pledge within certain nationalist circles points to the significance of Yasukuni beyond being a war memorial. While for the vast majority of its visitors it is a site at which to pray for ancestors who died in the service of Japan, for others it has become a way to deliberately provoke and strike out at China, at South Korea and at Japan’s own pacifist majority.

This is not how Yasukuni Shrine was envisioned at the outset. Originally established by the order of the Meiji Emperor in 1869 to commemorate the war dead of the conflicts which ended the Shogunate and created modern Japan, its role has expanded to cover the commemoration of almost 2.5 million named soldiers who died during various wars (at the main shrine), all of those who have died in the service of Japan, including non-Japanese nationals (at the Honden building), and all victims of the Second World War, regardless of affiliation or nationality (at the Chinreisha building).

In that regard, Yasukuni is not dissimilar to a national war memorial like Arlington Cemetery in Washington. The vast majority of Japanese people who visit Yasukuni do so for the same reason that Americans visit Arlington; they come to pay their respects to family members who died in the service of their country (however misguided their country’s aims may have been at that time).

Yasukuni’s contested political role arises from its crucial differences from Arlington. The post-war Constitution of Japan created a fairly strict separation of Church and State – or in this case, Shrine and State – which meant that Yasukuni Shrine could no longer be a state war memorial. The occupation authorities originally planned to raze the shrine entirely, but were persuaded to keep it by the intervention of the Roman Catholic Church, so it was handed a private religious corporation. This has led to a complex situation wherein neither the government nor the Emperor can exercise control over the nation’s most important and internationally recognised war memorial.

The lack of official state control was largely unimportant until the late 1970s, when one Matsudaira Nagayoshi took over as chief priest of the shrine. Matsudaira was a historical revisionist who rejected the verdict of the Tokyo War Crimes Tribunal and took it upon himself to add (“enshrine”) the names of all 14 convicted class-A war criminals at the shrine in a secret ceremony in 1978.

Matsudaira retired in 1992 and died in 2005, but his influence on Yasukuni remains powerful and damaging. The Showa Emperor refused to visit the shrine in the wake of Matsudaira’s appointment and the secret enshrinement of the war criminals. His son, the present Emperor, has taken the same stance, and no member of the Imperial Family has visited the shrine – which lies only minutes away from the palace – since 1975. Many Prime Ministers have also chosen to avoid Yasukuni, especially in the wake of harsh criticism from China when Prime Minister Nakasone visited in 1985. Given the legal separation of state from religion, Japan’s symbolic and actual leaders are powerless to intervene in affairs at the shrine or demand the removal of the war criminals from the shrine’s registers (which its religious authorities insist is impossible). For the past thirty years, most leaders have taken the only option remaining to them – snubbing Yasukuni entirely.

The influence of Matsudaira and of the revisionists whose reign at Yasukuni he ushered in is also felt in physical form. The shrine’s grounds house controversial memorials that directly challenge the established historical narrative of the war and the guilt of Japan’s convicted war criminals. Chief among them is the Yushukan – a war museum which is an exercise in dichotomy, with genuinely powerful exhibits from the war being grotesquely undermined by accompanying text and interpretation that has one foot in fantasy and the other in farce.

Given this, it’s not hard to see how official visits from government ministers inflame tensions with Japan’s neighbours, whose people were the victims of the war criminals enshrined there and whose suffering is deliberately questioned and erased by the childishly fantastical reimagining of history in the Yushukan. Cognisant of that, and either wiser or more capable of listening to good advice than he’s often given credit for, current Prime Minister Abe Shinzo has steered clear of Yasukuni Shrine since 2013. Other cabinet ministers and members of the Diet have been less circumspect; this year, Olympics Minister Marukawa Tamayo and Communications Minister Takaichi Sanae (previously noted on this blog for her threats to shut down broadcasters who don’t toe the government line) visited, as did former Defense Minister Nakatani Gen. It’s not only LDP ministers who visit Yasukuni; there is a cross-party group of MPs who lobby for politicians to make official trips to the shrine, and among this year’s August 15th visitors was Democratic Party leadership hopeful Nagashima Akihisa.

Other Diet members and ministers made private visits earlier, or will do so later. Criticism of those private visits is somewhat distasteful; whatever else Yasukuni has come to symbolise, it remains a place at which countless Japanese people, including Diet members, pray for departed ancestors and to give thanks to millions of people who gave their lives for the nation. It is important to draw a line between those who visit for private moments of worship and those who arrive with pomp, insist that their visit is official rather than personal, and make certain the cameras are waiting. Michael Cucek rightly describes this contrast as being between those who visit out of reverence, and those who visit out of a desire to transgress. If it seems to be in terribly bad taste to use a shrine commemorating a nation’s war dead and enshrining the relatives of millions of Japanese people simply as a way to jam one’s thumb in the eye of neighbours with whom you don’t get along, well, that’s because it absolutely is.

This is not to say that China and South Korea are blameless in how this dispute has developed. Both countries are guilty of stirring up national anger over Japan and wartime history in order to deflect attention from various failures of their own governments. There’s a long, long tradition of this in the post-war era. The Communist Party in China has always emphasised and on occasion enhanced Japanese wartime brutality not least in order to draw attention from its own brutality in the years after the war. South Korea’s post-war military dictatorship quietly took reparation money from Japan without informing its populace or distributing it to victims for whom it was intended, instead teaching its citizens that Japan had never apologised or paid reparations. In the case of both nations, matters of wartime history are made even more murky by the promotion of versions of history that, while closer to the truth than those of Japan’s historical revisionists, remain problematic and one-sided.

This all points to the fundamental problem with Yasukuni, with August 15th and with the whole question of how the war and its remembrance feeds into East Asian geopolitics. The problem is that almost none of this is actually about the war, or about history. It’s about contemporary issues; it’s about the fear, in Japan, of a declining nation thrown into sharp relief by the rise of China. It’s about the fear in both South Korea and China of an end to decades of rapid economic growth, and the prospect of a future not unlike Japan’s lost decades. It’s about concerns about political stability and national identity, and the utility of an external foe to focus attention away from stagnation and social problems at home. Each of the three governments shares some unequal portion of the blame for using history not as a way to establish fact, and remembrance not as a way to learn from the past and avoid its mistakes, instead using both as tools to achieve cynical, short-term political ends.

Yasukuni itself, however, remains an internal Japanese problem. The duality of its nature, simultaneously a legitimate place of worship and commemoration and a site for transgression and right-wing peacocking, makes it a thornier problem than many observers admit. Suggestions that the nearby Chidorigafuchi National Cemetery, a state-operated and much less controversial memorial, should replace Yasukuni as the focus for remembrance are simplistic and slightly naive. They misunderstand the differing roles of the memorials; the secular Chidorigafuchi is a “Tomb of the Unknown Soldier”, honouring some 350,000 soldiers whose remains could not be identified. The religious Yasukuni is a much more broad-ranging memorial and, crucially, enshrines the specifically named souls of some 2.5 million people. Removing Yasukuni from the nation’s rituals of war memory is an unreasonable demand. Expecting neighbouring countries to smile and nod at the deliberate provocation of politicians acting in an official capacity is equally unreasonable.

The “solution”, if any such thing can be achieved, will be a fudged, unofficial compromise – a return to a status quo in which nothing has actually been solved, but Japanese governments put their senior officials on shorter leashes, while Chinese and Korean authorities mute the tone of their statements. There’s some evidence of movement in that direction over the past couple of years, of a slow de-escalation of rhetoric and provocation around Yasukuni. Given time to bed in, perhaps such a compromise will allow Japanese people to commemorate their lost relatives at Yasukuni without rude interference from their own nation’s right-wing fringe.