C++11 - Convert to/from UTF-8/wchar_t

These functions do work on Linux/Mac and I use them with the Boost.Filesystem API. It is similar to UTF8-CPP and NoWide but not as advanced. It’s more for easily converting between UTF-8 and the native wchar_t type.

  • Always treat std::string as a UTF-8 string
  • Use std::string internally for everything
  • Use wchar_t APIs on Windows whenever possible with widen
  • Convert from std::wstring (use narrow) when saving data to disk
1
2
3
4
5
6
7
8
9
10
11
12
13
#include <locale>
std::string narrow(const std::wstring & wstr)
{
std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> convert;
return convert.to_bytes(wstr);
}
std::wstring widen(const std::string & str)
{
std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> convert;
return convert.from_bytes(str);
}

Change the locale inside main() so Boost.Filesystem works correctly on Windows.

1
2
3
4
#if defined(_WIN32)
std::locale::global(boost::locale::generator().generate(""));
boost::filesystem::path::imbue(std::locale());
#endif



Example on Windows with the CreateFileW API:

1
2
const std::string path = "my file.txt";
CreateFileW(widen(path).c_str(), GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ, nullptr, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);